Qwen2

Qwen2 AI

Qwen2 AI

Qwen2 AI

Qwen2 AI

Qwen2 AI

Qwen2 AI

Qwen2 AI

Qwen2 AI

Qwen2 AI

Qwen2 AI

My experience using Claude code with Local Llm, and full guide on how to set it up

Wanted to share a workflow I tested on a real flight, in case anyone else is trying to set up offline Claude Code. The core idea: using ollama to pull the needed model of what you need, and then use it to run claude code The setup, in order: Pull a model on home wifi the night before. `ollama pull ` — ~9 GB for a 14B, ~17 GB for a 26B. Don't try this at the gate. In Claude Code, point at Ollama. The cleanest path I found is wrapping it in two aliases: alias claude-local='ollama launch claude --model gemma4:26b' alias claude-cloud='claude' Verify on the ground with wifi physically off. If it works in airplane mode at home, it works at 10 km in the sky. Where I got it wrong: I prepped qwen2.5-coder:14b first because it's the model everyone recommends in local-LLM threads. On the flight, it choked on Claude Code's tool loop; one call took 25 seconds, another took 52. For a workflow that chains five or six tool calls per task, that's unusable. Switched mid-flight to gemma4:26b (which I'd pulled as a backup). Different category of model, RL-trained for tool use, not just code completion. The tool loop ran at a usable speed. The gap analysis I was running on a real codebase has been completed. Honest scorecard: ~70% of my normal Claude Code workflow worked on gemma4:26b offline. The 30% that didn't was heavy whole-repo reasoning When to reach for which: claude-local: no network, privacy-sensitive code (NDA / client work), drafting prompts before spending cloud tokens claude-cloud: multi-tool agentic work with subagents and MCP servers, whole-repo refactors, anything shipping to production Things that broke or surprised me: - Tool use is the weak point on local models; even good ones are less reliable at chaining many tool calls than cloud Claude - Battery drains noticeably faster while running a 26B with editor + browser open - Ollama's endpoint shape isn't 100% identical to Anthropic's. If you hit a strange parsing error mid-stream, that's usually why, and claude-cloud is the fix in the moment If anyone else has tested local models for Claude Code specifically (not Cursor, the loops are different), curious which models you've landed on. Wrote up the full thing in my newsletter, link if anyone wants the model-picker matrix + the verification checklist I use before flying: https://codemeetai.substack.com/p/how-i-run-claude-code-offline-the submitted by /u/MaterialAppearance21 [link] [comments]

Opus 4.6/4.7 regression is real and getting worse — 3 weeks of documented failures on a complex project, and a competing AI caught the mistakes Claude missed [long post]

I've been running Claude Pro (Opus 4.7 / Sonnet 4.6) for about 3 weeks on a complex personal AI infrastructure project. I keep structured session logs with timestamps and Birkenbihl-style metacognitive fields after every session. This is not anecdotal — I have receipts. The project for context I'm building a local persistent AI memory stack called GSOC Brain: Qdrant vector DB (~397K vectors across 11 source tags), Neo4j graph (123 nodes / 183 edges), Graphiti 0.29 entity extraction, Ollama with qwen2.5:14b + nomic-embed-text — all running natively on a Windows host. The system is supposed to give Claude cross-chat memory via a custom MCP server. On top of that, I'm operating 18+ custom skill files that define behavior rules for Claude across domains (OSINT/forensics, legal, content, infrastructure). The system prompt explicitly describes the full architecture on every session start. This is not a "chat with Claude" use case. This is sustained agentic work across multiple tools, multiple sessions, strict context requirements, and high-stakes outputs (including legal document drafts). Bug 1: Token overconsumption since update 2.1.88 (late March 2026) Opus 4.7 started burning daily usage limits at a completely different rate after an update around March 31. In one session I hit 94% of my daily limit within approximately 4 messages. The boot sequence — fetching context from Notion MCP, searching past sessions, loading memory — consumed what felt like 10–20x the previous token rate. GitHub issues #42272, #50623, and #52153 document identical patterns from other users. The model appears to over-generate internally even for simple responses. End result: I had to switch to Sonnet 4.6 for most productive work because Opus 4.7 is simply unusable under the daily limit. Bug 2: Claude Code Desktop App completely broken (reported May 14, Conv. 215474208295333) The Desktop App hangs on every single input. Including typing "hello" with no files. Reproducible across: Sonnet 4.6 and Opus 4.7 Multiple fresh sessions With and without u/file references After full reinstall The VS Code extension works fine. Only the Desktop App is broken. Reported May 14. No fix, no acknowledgment. Bug 3: Platform / context confusion — 5 documented errors in a single session, chat aborted On April 29, I had to formally abort an Opus 4.7 session and hand off to Opus 4.6 after documenting 5 consecutive errors. The session log entry literally reads "Opus 4.7 Abbruch (5 Fehler): Zeitrechnung, Platform-Verwechslung, falsche Schlüsse": Miscalculated the current time despite being told the exact time Insisted the Brain stack was running on a Linux VM (BURAN) — the system prompt and memory both explicitly stated C:\gsoc-brain on Windows Drew false inferences from backup file paths rather than the stated architecture Contradicted the stated platform in the same response it had just received Confused WebClaude and Desktop Claude capability boundaries These aren't edge cases. The architecture was in the system prompt, in memory, and in the injected Notion context. Opus 4.7 ignored all of it. Bug 4: Skill files ignored in production I maintain 18+ custom skill files loaded into the system prompt. These include explicit hard rules — e.g., "activate keilerhirsch-knowledge skill for ALL architecture decisions, web search is not optional." In the session that caused the Docker-to-Native migration disaster, I later wrote in my own session log: The model proceeded to recommend outdated tools from training data rather than searching current documentation. It recommended NSSM (last meaningful update 2017) as a Windows service wrapper. NSSM is dead. A competing AI caught this immediately. Bug 5: Another AI caught what Claude missed in a single pass This is the part that stings most. When the Docker-based Brain setup kept failing, I fed the architecture docs into another AI (Manus) for a deep audit. In one pass it identified 5 critical corrections that Claude had never caught across weeks of sessions: NSSM is dead since ~2017 → correct replacement is WinSW or Servy Neo4j 2025.01+ requires Java 21 — Claude had never flagged this, the services kept failing silently Qdrant needs Windows file-handle-limit adjustments to run reliably Orphaned vector risk between Qdrant ↔ Neo4j without a Tentative-Write pattern in the save operation BGE-M3 embeddings (MTEB 63.2, 8192 token context) as a better alternative to nomic-embed-text My own session log the next day reads: Claude was answering from stale training data. The skill that explicitly says "don't do this" was being ignored. Another AI caught it in round one. Bug 6: MCP Server 20-minute Neo4j hang — still unresolved After the native migration, the custom gsoc_mcp_server.py developed a reproducible hang of exactly ~20 minutes between Qdrant connect and Neo4j connect on every startup. Log timestamps from 4 consecutive restarts: 14:59 → 15:20 (21 min) 15:29 → 15:51 (22 min)

ai slop? who knows~

I investigated whether routing a transformer's forward activations through a lossy Dual E8 (E16) lattice bottleneck and injecting them back into the residual stream is viable, and where the boundary of generative stability lies. **The core finding:** There is a sharp empirical stability threshold at a blend ratio of $\beta = 0.20$. Beyond this boundary, open-ended generation collapses into semantic loops and repetition lock. --- ### The Mechanism Standard LLM states are high-dimensional floats. Rather than applying traditional scalar quantization (like INT4), I mapped high-dimensional activations onto a conceptual torus via a sinusoidal map and projected them onto Dual E8 lattice hemispheres. Full replacement of MLP layers with geometric bottlenecks universally collapsed the model. Instead, I implemented a residual blend: $$\text{out} = (1-\beta)\cdot\text{original} + \beta\cdot\text{geometric}$$ --- ### The $\beta = 0.20$ Sweep (Qwen2.5-0.5B) Sweeping $\beta$ from 0.10 to 0.50 across layers 8–13 of `Qwen2.5-0.5B` reveals a sharp phase transition: * **$\beta \ge 0.25$** : Generation succumbs to heavy repetition pressure and semantic drift. The geometry acts as an attractor, trapping the decoding process ("loop-lock"). * **$\beta = 0.20$** : The stability boundary. This is the highest injection ratio of lossy geometric signal that maintains both numerical activation fidelity (Avg Cosine > 0.99) and open-ended generation quality (low repeated n-grams). * **$\beta \le 0.10$** : The perturbation is largely absorbed and damped by the transformer's layer normalizations, making the intervention invisible. Here is the data from a 300-iteration sweep: | $\beta$ | Min Cosine | Avg Cosine | Max MSE | Rep-3g (Repetition Rate) | | :--- | :--- | :--- | :--- | :--- | | 0.10 | 0.9972 | 0.9979 | 0.0024 | 0.134 | | **0.20** | **0.9907** | **0.9916** | **0.0106** | **0.093** | | 0.25 | 0.9839 | 0.9865 | 0.0171 | 0.084 | | 0.30 | 0.9648 | 0.9771 | 0.0255 | 0.190 | | 0.50 | 0.9171 | 0.9288 | 0.0850 | 0.412 | Semantic scoring (evaluating prompt relevance and similarity to the unmodified baseline): | $\beta$ | Avg Cosine | Rep-3g | Relevance | Patched-to-Baseline Sim | | :--- | :--- | :--- | :--- | :--- | | 0.10 | 0.9980 | 0.223 | 0.781 | 0.889 | | **0.20** | **0.9918** | **0.075** | **0.752** | **0.854** | | 0.25 | 0.9871 | 0.232 | 0.717 | 0.801 | | 0.30 | 0.9760 | 0.392 | 0.725 | 0.764 | --- ### Generalization (1.5B & 3B Models) The $\beta = 0.20$ boundary generalizes across larger model sizes (`Qwen2.5-1.5B` and `Qwen2.5-3B` in 4-bit) on the activation-cosine axis: | Model | $\beta$ | Min Cosine | Avg Cosine | Max MSE | Rep-3g | | :--- | :--- | :--- | :--- | :--- | :--- | | **1.5B** | 0.10 | 0.9988 | 0.9989 | 0.0027 | 0.267 | | | **0.20** | **0.9862** | **0.9939** | **0.0105** | **0.128** | | | 0.25 | 0.9904 | 0.9919 | 0.0166 | 0.398 | | | 0.30 | 0.9733 | 0.9815 | 0.0235 | 0.307 | | | 0.40 | 0.9368 | 0.9551 | 0.0487 | 0.191 | | **3B (4-bit)** | 0.10 | 0.9964 | 0.9976 | 0.0122 | 0.033 | | | **0.20** | **0.9861** | **0.9904** | **0.0455** | **0.115** | | | 0.25 | 0.9604 | 0.9799 | 0.0654 | 0.043 | | | 0.30 | 0.9702 | 0.9778 | 0.0987 | 0.050 | | | 0.40 | 0.9158 | 0.9390 | 0.1728 | 0.025 | *Note: In the 3B model, repetition pressure remained low across all sweeps, but the validation cosine degraded identically at $\beta \ge 0.25$.* I also tested layer-level oscillating $\beta$ schedules (e.g., sine waves across layers), but they degraded open-ended text quality compared to a fixed, constant injection ratio. --- ### Storage Compression Prototypes Utilizing the Dual E8/E16 lattice as a computational substrate also yields high theoretical storage efficiency in early prototypes: 1. **KV Cache (8$\times$)** : FP16 KV cache compressed to INT8 coordinates, reducing footprint from 0.21 MB to 0.02 MB. 2. **Weights (112$\times$)** : Projected a dense $[4864, 896]$ MLP weight matrix down to a 0.07 MB E16 footprint. (Cosine similarity of the uncalibrated weight matrix multiplication was limited to $\sim$0.078, indicating that Quantization-Aware Training is mandatory for parameter viability). A **pre-projected decompression bypass** was designed to run matrix multiplications directly against lattice coordinates without upcasting, avoiding memory bandwidth bottlenecks. --- ### Policy Constraints (Negative Result) I evaluated whether residual E16 projection could act as a steering substrate to enforce safety policies. It cannot. While $\beta = 0.20$ preserves generation quality, the lossy nature of E16 projection strips out the logical nuances required to maintain strict boundaries. Dedicated supervised control heads remain necessary. --- ### Implications & Next Steps Snapping post-training activations to a fixed algebraic lattice is ultimately lossy. The real frontier here is **native geometric transformers** —designing and training networks from scratch with E8/E16 constraints native to both weight matrices and activation routing. submitt

eTPS Site Plan – Simple Leaderboard + What You’ll Actually See

Building on the last post, here’s what the first version of effectiveTPS will look like. **Core display (v1):** - Clean table comparing popular local models - Raw TPS (the marketing number everyone shows) - eTPS (the new metric that actually measures useful output in real conversations) - Time to First Token (how long you wait before it starts replying) - Effectiveness Index = (eTPS ÷ Raw TPS) × 100 — higher is better **Example leaderboard (early test data):** | Model | Raw TPS | eTPS | Time to First Token | Effectiveness Index | |--------------------|---------|--------|---------------------|---------------------| | Llama 3.1 70B | 45.2 | 38.7 | 1.4s | **86** | | Qwen2.5-32B | 68.4 | 52.1 | 0.8s | **76** | | Gemma 2 27B | 71.3 | 44.6 | 0.6s | **63** | I’ve been running these tests through a structured multi-turn analysis framework I built to evaluate complex workflows. That’s how eTPS was stress-tested — not just single-turn benchmarks, but real back-and-forth sessions. Advanced mode (toggle) will add latency percentiles, cost-per-quality, and consistency scoring later. For v1 the goal is to keep it dead simple and immediately useful, even if you’re not deep into AI. The whole point is to cut through the noise and show which models actually deliver useful work, not just raw speed. What do you think should be added (or removed) for the first version? Any metrics you’d want to see front-and-center? **TL;DR:** Simple leaderboard with Raw TPS, eTPS, Time to First Token, and a clear Effectiveness Index. Advanced stuff stays hidden until you want it. Feedback welcome. submitted by /u/axendo [link] [comments]

[P] QLoRA Fine-Tuning of Qwen2.5-1.5B for CEFR English Proficiency Classification (A1–C2) [P]

I fine-tuned Qwen2.5-1.5B for multi-class CEFR English proficiency classification using QLoRA (4-bit NF4). The goal was to classify English text into one of the 6 CEFR levels (A1 → C2), which can be useful for: adaptive language learning systems, placement testing, readability estimation, educational NLP applications. Dataset The dataset contains 1,785 English texts balanced across: 6 CEFR levels, 10 domains/topics. The samples were synthetically generated using: Groq API Llama-3.3-70B Generation constraints were designed to preserve: vocabulary complexity, grammatical progression, sentence structure variation, CEFR-specific linguistic patterns. Training Setup Base model: Qwen2.5-1.5B Fine-tuning method: QLoRA 4-bit NF4 quantization LoRA adapters Only ~0.28% of model parameters were trained. Results Held-out test set: 179 samples Metrics: Accuracy: 84.9% Macro F1: 84.9% Per-level recall: Level Recall A1 96.6% A2 90.0% B1 90.0% B2 86.7% C1 86.7% C2 60.0% Most errors come from C1/C2 confusion, which is expected due to the subtle linguistic boundary between those levels. Deployment I also built: a FastAPI inference API, Docker deployment setup. Example Usage from transformers import AutoModelForSequenceClassification, AutoTokenizer import torch model = AutoModelForSequenceClassification.from_pretrained( "yanou16/cefr-english-classifier" ) tokenizer = AutoTokenizer.from_pretrained( "yanou16/cefr-english-classifier" ) text = "Artificial intelligence is transforming many industries." inputs = tokenizer(text, return_tensors="pt") with torch.no_grad(): outputs = model(**inputs) pred = outputs.logits.argmax(dim=-1).item() print(pred) Feedback is welcome, especially regarding: evaluation methodology, synthetic data quality, improving C2 classification performance, better benchmarking approaches. submitted by /u/Professional-Pie6704 [link] [comments]

A Hackable ML Compiler Stack in 5,000 Lines of Python [P]

Hey r/MachineLearning, The modern ML (LLM) compiler stack is brutal. TVM is 500K+ lines of C++. PyTorch piles Dynamo, Inductor, and Triton on top of each other. Then there's XLA, MLIR, Halide, Mojo. There is no tutorial that covers the high-level design of an ML compiler without dropping you straight into the guts of one of these frameworks. I built a reference compiler from scratch in ~5K lines of pure Python that emits raw CUDA. It takes a small model (TinyLlama, Qwen2.5-7B) and lowers it to a sequence of CUDA kernels through six IRs. The goal isn't to beat Triton; it is to build a hackable, easy-to-follow compiler. Full article: A Principled ML Compiler Stack in 5,000 Lines of Python Repo: deplodock The pipeline consists of six IRs, each closer to the hardware than the last. Walking the following PyTorch code through every stage (real reference compiler output with names shortened for brevity and comments added): torch.relu(torch.matmul(x + bias, w)) # x: (16, 64), bias: (64,), w: (64, 16) Torch IR. Captured FX graph, 1:1 mirror of PyTorch ops: bias_bc = bias[j] -> (16, 64) float32 add = add(x, bias_bc) -> (16, 64) float32 matmul = matmul(add, w, has_bias=False) -> (16, 16) float32 relu = relu(matmul) -> (16, 16) float32 Tensor IR. Every op is decomposed into Elementwise / Reduction / IndexMap. Minimal unified op surface, so future frontends (ONNX, JAX) plug in without touching downstream passes: bias_bc = bias[j] -> (16, 64) float32 w_bc = w[j, k] -> (16, 64, 16) float32 add = add(x, bias_bc) -> (16, 64) float32 add_bc = add[i, j] -> (16, 64, 16) float32 prod = multiply(add_bc, w_bc) -> (16, 64, 16) float32 red = sum(prod, axis=-2) -> (16, 1, 16) float32 matmul = red[i, na, j] -> (16, 16) float32 relu = relu(matmul) -> (16, 16) float32 The (16, 64, 16) intermediate looks ruinous, but it's never materialized; the next stage fuses it out. Loop IR. Each kernel has a loop nest fused with adjacent kernels. Prologue, broadcasted multiply, reduction, output layout, and epilogue all collapse into a single loop nest with no intermediate buffers. === merged_relu -> relu === for a0 in 0..16: # free (M) for a1 in 0..16: # free (N) for a2 in 0..64: # reduce (K) in0 = load bias[a2] in1 = load x[a0, a2] in2 = load w[a2, a1] v0 = add(in1, in0) # prologue (inside reduce) v1 = multiply(v0, in2) acc0 <- add(acc0, v1) v2 = relu(acc0) # epilogue (outside reduce) merged_relu[a0, a1] = v2 Tile IR. The first GPU-aware IR. Loop axes get scheduled onto threads/blocks, Stage hoists shared inputs into shared memory, and a 2×2 register tile lets each thread accumulate four outputs at once. The K-axis is tiled into two outer iterations of 32-wide reduce. Three-stage annotations below carry the heaviest optimizations: buffers=2@a2 — double-buffer the smem allocation along the a2 K-tile loop, so loads for iteration a2+1 overlap compute for a2. async — emit cp.async.ca.shared.global so the warp doesn't block on global→smem transfers; pairs with commit_group/wait_group fences in Kernel IR. pad=(0, 1, 0) — add 1 element of padding to the middle smem dim so warp-wide loads don't all hit the same bank.kernel k_relu_reduce Tile(axes=(a0:8=THREAD, a1:8=THREAD)): for a2 in 0..2: # K-tile # meta: double-buffered, sync (small, no async needed) bias_smem = Stage(bias, origin=((a2 * 32)), slab=(a3:32@0)) buffers=2@a2 ​ kernel k_relu_reduce Tile(axes=(a0:8=THREAD, a1:8=THREAD)): for a2 in 0..2: # K-tile bias_smem = Stage(bias, origin=((a2 * 32)), slab=(a3:32@0)) buffers=2@a2 x_smem = Stage(x, origin=(0, (a2 * 32)), slab=(a0:8@0, a3:32@1, cell:2@0)) pad=(0, 1, 0) buffers=2@a2 async w_smem = Stage(w, origin=((a2 * 32), 0), slab=(a3:32@0, a1:8@1, cell:2@1)) buffers=2@a2 async # reduce for a3 in 0..32: in0 = load bias_smem[a2, a3] in1 = load x_smem[a2, a0, a3, 0]; in2 = load x_smem[a2, a0, a3, 1] in3 = load w_smem[a2, a3, a1, 0]; in4 = load w_smem[a2, a3, a1, 1] # prologue, reused 2× across N v0 = add(in1, in0); v1 = add(in2, in0) # 2×2 register tile acc0 <- add(acc0, multiply(v0, in3)) acc1 <- add(acc1, multiply(v0, in4)) acc2 <- add(acc2, multiply(v1, in3)) acc3 <- add(acc3, multiply(v1, in4)) # epilogue relu[a0*2, a1*2 ] = relu(acc0) relu[a0*2, a1*2 + 1] = relu(acc1) relu[a0*2 + 1, a1*2 ] = relu(acc2) relu[a0*2 + 1, a1*2 + 1] = relu(acc3) Kernel IR. Schedule materialized into hardware primitives. THREAD/BLOCK become threadIdx/blockIdx, async Stage becomes Smem + cp.async fill with commit/wait fences, sync Stage becomes a strided fill loop. Framework-agnostic: same IR could lower to Metal or HIP: kernel k_relu_reduce Tile(axes=(a0:8=THREAD, a1:8=THREAD)): Init(acc0..acc3, op=add) for a2 in 0..2: # K-tile Smem bias_smem[2, 32] (float) StridedLoop(flat = a0*8 + a1; < 32; += 64): bias_smem[a2, flat] = load bias[a2*32 + flat] Sync # pad row to 33 to kill bank conflicts Smem x_smem[2, 8, 33, 2] (float) StridedLoop(flat = a0*8 + a1; < 512; += 64): cp.async x_smem[a2, flat/64, (flat/2)%32, flat%2] <- x[flat/64*2 + flat%2, a2*3

Speculative Decoding Implementations: EAGLE-3, Medusa-1, PARD, Draft Models, N-gram and Suffix Decoding from scratch [P]

I’ve been working on an educational implementation repo for speculative decoding: https://github.com/shreyansh26/Speculative-Decoding The goal is not to wrap existing libraries, but to implement several speculative decoding methods from scratch behind a shared decoding/evaluation contract so that the differences between proposer designs are easier to study. Implemented methods so far: EAGLE-3 Medusa-1 standard draft model speculation PARD / parallel draft models n-gram prompt lookup suffix decoding The repo has both training and inference paths where applicable. For learned proposers, I use Qwen/Qwen2.5-7B-Instruct as the target model and small learned/speculative heads or draft models, depending on the method. For training-free methods, the proposer is built from the prompt/generated context. A few things I wanted the repo to make explicit: The distinction between proposer quality and verifier cost. Why a high acceptance rate does not always imply higher throughput. Why methods like PARD can be faster despite lower acceptance than an autoregressive draft model. How EAGLE/Medusa-style learned heads differ from draft-model speculation. How simple methods like n-gram and suffix decoding behave when the prompt contains a reusable structure. The repo includes benchmark summaries, command lines, checkpoints/exports, and implementation notes. Some results are intentionally on small train-overlap eval slices due to compute constraints, so I would treat the numbers as implementation/behavioral benchmarks rather than broad generalization claims. I built this mostly as a learning resource for people who want to understand speculative decoding at the algorithm + systems boundary: how the proposer is trained, how draft tokens are generated, how target verification works, what gets cached, and where the speedups actually come from. submitted by /u/shreyansh26 [link] [comments]

Seashell: open source MCP that bridges Claude and Wave Terminal, no API key needed

https://preview.redd.it/hagoyq9jrbxg1.png?width=1024&format=png&auto=webp&s=6be8dba9abafe3c7eb113392ec9ca4aeb7b46fa9 Hey folks, So I've been working on this thing called Seashell for a few weeks. It started because I was tired of treating Claude Desktop and Wave Terminal like two unrelated apps. Yes, they technically share session storage on disk, but neither side really knows about the other. I'd be deep in a Claude conversation in Desktop, drop into a terminal to check something, and the whole context was just gone. Seashell fixes that. It's an open source MCP server that plugs into Claude Desktop (Chat, Cowork, Code, all of them) and the claude CLI, plus a small set of friendly shell commands that tie everything together. The bridge goes both ways. From Claude, you can fully configure and operate Wave Terminal through MCP tools (settings, widgets, blocks, themes, scrollback, the works). From Wave, you can resume any past Claude session by name, leave Claude notes, ask async questions, all of it. And one thing worth flagging up front, because it tripped me up with other MCPs: Seashell rides on your existing Claude subscription. No API key from console.anthropic.com, no per token billing layered on top. If you already have Claude Pro or Max, there's nothing extra to pay. It just uses the same auth you set up with claude auth login. Here's the marquee feature in action: $ hey continue with myapp 🔄 Resuming session a1b2c3d4 (project: myapp)... > What's the latest on the auth refactor? We finished extracting AuthService. Tests pass. Next is wiring it into the API layer. I have a draft in routes/auth.py on line 142. That's a fresh terminal. Zero typing of session IDs, same conversation history as last night's Claude session. Fuzzy matching is built in too, so even something verbose like "hey continue with the auth refactor I was doing yesterday" figures it out. A few other things you can do with it: Configure Wave Terminal entirely from Claude, in any mode. Ask Claude in Chat to set your terminal theme, add a fish widget, switch to dracula, whatever. The full Wave config surface is exposed as MCP tools. Leave Claude a note from any terminal with seashell-msg "fix the build error" and Claude reads it next time you chat Ask Claude something async with seashell-ask "what was the last refactoring decision we made?". It blocks until you get a reply, which is great when Desktop is busy on a different conversation Run seashell-mirror-mcp to sync all your Desktop MCP servers (Trello, GitHub, Slack, whatever you have wired up) over to the CLI in one go, so resumed sessions never lose their tools Read another session's transcript directly from Claude, useful for "what's the status of project X?" type questions across projects The whole spirit of this thing is open source on open source. Wave Terminal is open source. Seashell is open source. They sit on top of each other and work together. That's the kind of stack I want to be using and contributing to. Who is this actually for? macOS users right now (Linux and Windows are not tested yet) Folks who use Claude Desktop AND the claude CLI and want them to feel like one tool instead of two Wave Terminal users (Seashell is tightly integrated, but it also works fine without Wave) fish shell users (there's a polished bundle with Alt+E editor mode, qwen2.5-coder powered natural language routing, daylight widget, the lot) Honest caveats so nobody is surprised: I built it for my own workflow first and polished afterwards, so bugs are going to happen Claude session continuity works for Code mode plus the claude CLI, since they share the same .jsonl storage. Desktop's Chat and Cowork keep their transcripts internal, so those modes are not resumable from a terminal The SessionStart hook fires only for interactive claude invocations, not for claude -p. Filesystem discovery covers most of that gap MIT licensed. Pull requests and issues are very welcome. If you have ideas for what's missing, I'd love to hear them. GitHub: https://github.com/M-Pineapple/seashell Enjoy 🍍 submitted by /u/CryptBay [link] [comments]

I Built a desktop app for generating LLM fine-tuning datasets — started it a week ago while learning FT

Hey, I've been building side projects with Claude Code for a few months, but I'm completely new to fine-tuning — started experimenting maybe a week ago. From day one I wanted a GUI for the dataset side of the workflow, so this desktop app grew alongside my very first FT attempts. I know there are similar apps out there, but I wanted something simple that non-technical users could run with open-source models end-to-end. To sanity-check whether the datasets were actually useful I fine-tuned Qwen2.5-Coder-7B-Instruct on them and ran HumanEval / HumanEval+ (pass@1, 5 runs). Picked these benchmarks because they match the dataset's focus and run fast on my machine: I know it's not much but know now that app work :) - Base: 55.5% / 49.0% - FT V2 (1135 samples from the app): 60.0% / 54.0% Error bars don't overlap so it's at least not noise. Obviously HumanEval is only one slice — YMMV with other categories / criteria. https://reddit.com/link/1srz5aq/video/zubr426holwg1/player Stack: Next.js 16 + FastAPI + SQLite, packaged as standalone binary (Win/Linux). Code: https://github.com/AronDaron/dataset-generator Fine-tuned model: https://huggingface.co/AronDaron/Qwen2.5-Coder-7B-Instruct-DatasetGen-v2 Datasets: https://huggingface.co/datasets/AronDaron/dataset-gen-v1 / https://huggingface.co/datasets/AronDaron/dataset-gen-v2 Happy to hear feedback, especially if something doesn't work on your setup or if the approach misses something obvious — this is my first finetune llm tool release. submitted by /u/AronSan [link] [comments]

SGOCR: A Spatially-Grounded OCR-focused Pipeline & V1 Dataset [P]

Hello everyone! I've been independently researching & developing small-but-powerful vision-language models (VLMs) and noticed a gap in visual datasets - none were teaching my model to simply ground text in imagery, but trying to get it to reason about the text or about the scene itself. This lead me down a 2 week side-side-project to create SGOCR, an open source dataset pipeline for generating spatially-grounded, OCR-focused VQA tuples with tons of rich metadata to support diverse VLM training strategies. Code v1 dataset My development began with simply prompting Qwen2.5-VL locally and grew into a multi-stage beast. At one point, my OCR-stage looked for concensus between 3 text recognition models (Parseq), my anchor stage did the same between GroundingDino, Florence 2, and SAM 3.1, and verification required passes from both Gemini 3.1 Pro & ChatGPT 5.3 Codex to pass. I discovered that less is more in this case, and landed on using Nvidia's nemotron-ocr-v2 for text extraction, a combination of Gemma4 with a Qwen3-VL fallback for anchor discovery & labeling, and then gemini-2.5-flash as a teacher model with simple grounding checks for verification. I got away with using the smaller 2.5 Flash teacher model due to the highly grounded annotations provided in context allowing flash to focus on semantics. I utilized an agentic loop for development after first creating a dataset review frontend that would store my personal accept/reject/maybe marks to be referenced as human-grounded context later. I bootstrapped this process into a quality score that reflected the aspects of questions I accepted, and from there the rest was much easier to automate. I run a custom optimization loop agent, based on Karpathy's autoresearch (which I found a bit too hyperparameter-searchy), that uses a sweep-based process that allows better holisitc observation, an oppurtunity to make code changes, and less risks of good ideas dying earlier due to their evals being slightly less than another variant's. I'm looking for general feedback and interested if other people were looking for something like this, or building similar VLMs. Thanks for reading! submitted by /u/Dreeseaw [link] [comments]

Converting XQuery to SQL with Local LLMs: Do I Need Fine-Tuning or a Better Approach? [P]

​ I am trying to convert XQuery statements into SQL queries within an enterprise context, with the constraint that the solution must rely on locally run LLMs. A key challenge is the limited availability of training data (pairs of XQueries and their corresponding SQL queries), especially with enough diversity to cover different patterns. I initially experimented with a parsing-based approach. The idea was to extract elements such as table names, columns, and conditions from the XQuery (using a Python script), map them to SQL components, and pass this structured representation to an LLM. However, this approach depended heavily on regex-based parsing and broke down when the input queries varied in structure. I then tried a prompt-engineering approach, defining strict rules and templates for how SQL queries should be generated. While this worked to some extent for simpler inputs, the outputs became inconsistent and often incorrect for more complex or longer XQueries. At the moment, I am considering fine-tuning a local LLM using PEFT (QLoRA) with a Qwen2.5-Coder 7B model. However, the dataset available is quite small (\~110–120 samples) and not very diverse. The main issues observed so far: Sensitivity to variations in how XQueries are written. Missing conditions or columns in generated SQL for longer inputs. Given these constraints, I am trying to understand the most effective direction to take. Would fine-tuning with such limited data be sufficient, or are there better approaches for handling this kind of structured query translation problem? Happy to provide more details if needed. submitted by /u/genius03noob [link] [comments]

I built a local-first MCP server that gives Claude Code persistent memory, a knowledge graph, and a consent framework — and Claude is just the first client

I've been building this for a couple of years. It started as "what if my AI assistant actually remembered things," and it became something bigger. The short version: I built a local AI infrastructure layer that runs entirely on my machine. No cloud. No exposed ports. My data stays on my hardware. And this week it's finally at a point where I can share it. --- What it is willow-1.7 is a Model Context Protocol server. Claude Code connects to it at session start via stdio — no HTTP, no ports, no supervisor. A direct pipe. Through that connection, Claude gets 44 tools: - Persistent memory — a Postgres knowledge graph (atoms, entities, edges) that survives sessions - Local storage — SQLite per collection, with a full audit trail and soft-delete - Inference routing — local Ollama first, then Groq / Cerebras / SambaNova as free-tier fallback if Ollama is down - Task queue — Claude submits shell tasks to Kart, a worker that polls Postgres and executes them - SAFE authorization — every agent that wants knowledge graph access must present a GPG-signed manifest. No valid signature = access denied. Revoke an agent by deleting its folder. The filesystem is the ACL. - Session handoffs — structured handoff documents written to disk and indexed in Postgres, so the next session can pick up from where the last one ended --- The authorization model This part is unusual enough that it's worth explaining. Each application that wants to access the knowledge graph has a folder on a separate partition (/media/willow/SAFE/Applications/ /). That fo - safe-app-manifest.json — declares permissions and data streams - safe-app-manifest.json.sig — a GPG detached signature of the manifest On every access attempt, the gate checks: folder exists → manifest present → signature present → gpg --verify passes. All four must pass. Any failure → deny + log. No code changes to revoke access. Delete the folder, and that agent is done. I've been running 17 AI professors through this gate for months. Each one has its own signed folder, its own permitted data streams, its own context. None of them can access data outside their declared scope. --- What powers it locally Ollama runs the inference. Currently using qwen2.5:3b as the default. The system routes there first and falls back to free cloud APIs only if Ollama is unavailable. But Claude is just the first client. The MCP server speaks stdio MCP. Any agent that understands the protocol can connect — Gemini, local models, anything. The longer plan: Yggdrasil. A small model trained on the operational patterns this system generates — session handoffs, ratified knowledge atoms, governance logs. When that model is trained, it replaces the cloud fleet entirely. The system becomes fully air-gappable. And after that: an open-source Claude Code equivalent. A terminal AI agent that boots from your local repo, connects to willow via stdio, and has no dependencies you don't control. No telemetry. No cloud account required. Just you and the tools you built. willow-1.7 is the bus everything else rides. The client is just the first thing attached to it. --- Why local-first matters to me I have two daughters. I'm building this so they grow up with tools that help them think instead of thinking for them. That don't own their journals. That don't optimize their attention. That expire when they close the app. The current model is: agree once, we own everything forever. Your notes train our models. Your data lives in our building. Local-first is the other way. Your data lives on your machine. Consent is session-based — the system asks every time, and that permission expires when you're done. If you walk away, it stops. --- The bootstrap There's a separate installer repo, willow-seed, that handles the full setup from scratch — clones the repo, creates the Postgres database, scaffolds the first SAFE agent entry, writes the MCP config. Stdlib only, no dependencies. Consent gates before every action. python seed.py That's it. Tested it this week on a fresh partition. It works. --- Links - willow-1.7: https://github.com/rudi193-cmd/willow-1.7 - willow-seed: https://github.com/rudi193-cmd/willow-seed - SAFE spec: https://github.com/rudi193-cmd/SAFE --- Happy to answer questions. Still building. ΔΣ=42 submitted by /u/BeneficialBig8372 [link] [comments]

Trained a Qwen2.5-0.5B-Instruct bf16 model on Reddit post summarization task with GRPO written from scratch in PyTorch - updates! [P]

So, yesterday run was a success and I did get an avg rollout length of about 64 tokens as attached in the image! This was with quality_reward + length_penalty (more info below!) Next, I'll be going with length penalty as the reward and with the mistake of counting characters as tokens fixed and see if there is any gaming the system stuff or degraded outputs! The rewards I used were 2: length_penalty : basically, -abs(response_length - MAX_LENGTH) quality_reward: ROUGE-L, which is basically LCS of golden summarizations I had as part of the above dataset, to ensure we have some structure throughout the responses generated Setup: 3x Mac Minis in a cluster running MLX. One node drives training using GRPO, two push rollouts via vLLM. Trained two variants: length penalty only (baseline) length penalty + quality reward (BLEU, METEOR and/or ROUGE-L ) Eval: LLM-as-a-Judge (gpt-5) Used DeepEval to build a judge pipeline scoring each summary on 4 axes: Faithfulness — no hallucinations vs. source Coverage — key points captured Conciseness — shorter, no redundancy Clarity — readable on its own and minimize degradation. https://preview.redd.it/7nrsulwdkbvg1.png?width=800&format=png&auto=webp&s=a3306b54ca63c6557534d9393b2d9b099c4b1b03 https://preview.redd.it/xlcnme2gkbvg1.png?width=800&format=png&auto=webp&s=57073ff1a9aea796d04aae5ef6d22fee1939d30b submitted by /u/East-Muffin-6472 [link] [comments]

Trained a Qwen2.5-0.5B-Instruct bf16 model on Reddit post summarization task with GRPO [P]

So, a few days back I shared a post where I trained a tiny Qwen2.5-0.5B-Instruct model on smoltldr (reddit post summarization dataset of 2k rows), to output summaries of about 64 max length using RLVR with GRPO . However, there was a catch! The wandb charts for avg response length was going down and saturated around 10-15 tokens on an avg. This was the result of me confusing between character counts and token counts, I meant to do 64 tokens but rather I accidentally went for 64 characters! Hence the charts showed a sharp decline and convergence towards a response length of on and off 15 tokens. The rewards I used were 2: length_penalty : basically, -abs(response_length - MAX_LENGTH) quality_reward: a ROUGE-L, which is basically LCS of golden summarizations I had as part of the above dataset, to ensure we have some structure throughout the responses generated and minimize degradation. Trained to one full epoch with a batch size of 2 max (before getting a OOM), the results were identical to the previous run, however, with one crucial difference - without a quality reward in my previous runs, the system tried to game the rewards by outputting stuff like "-------*20" tokens thats it! But not this time since I got the near same results for rewards of both the experiments when I included both vs just length penalty, and no degradation in the rollouts after 1 full epoch so I wonder why? Anyways, next up: Find out why GRPO didn't try other game the reward system? Try out metrics other than ROUGE-L to get better summarizations maybe Setup LLM-As-A-Judge to quantify the results. Train some HF SmolLM series now! What if I told in the prompt itself about the reward system and about the MAX_LENGTH with the task? Different MAX_LENGTH? https://preview.redd.it/mf7rux5lhyug1.png?width=800&format=png&auto=webp&s=bc54273f644ee2306b03834e037ab3e91f3b0582 https://preview.redd.it/1es4n61mhyug1.png?width=800&format=png&auto=webp&s=a8cc4249e646f03e8396cf79e640e27fcd1edfce https://preview.redd.it/djsslwsmhyug1.png?width=800&format=png&auto=webp&s=91589c746ac7a2c43d724e4768e8cb610288dee4 submitted by /u/East-Muffin-6472 [link] [comments]

KIV: 1M token context window on a RTX 4070 (12GB VRAM), no retraining, drop-in HuggingFace cache replacement - Works with any model that uses DynamicCache [P]

Been working on this for a bit and figured it was ready to share. KIV (K-Indexed V Materialization) is a middleware layer that replaces the standard KV cache in HuggingFace transformers with a tiered retrieval system. The short version: it keeps recent tokens exact in VRAM, moves old K/V to system RAM, and uses K vectors as a search index to pull back only the ~256 most relevant V entries per decode step. Results on a 4070 12GB with Gemma 4 E2B (4-bit): 1M tokens, 12MB KIV VRAM overhead, ~6.5GB total GPU usage 4.1 tok/s at 1M context (8-10 tok/s on GPU time), 12.9 tok/s at 4K 70/70 needle-in-haystack tests passed across 4K-32K Perfect phonebook lookup (unique names) at 58K tokens Prefill at 1M takes about 4.3 minutes (one-time cost) Decode is near-constant regardless of context length The core finding that makes this work: K vectors are smooth and structured, which makes them great search indices. V vectors are high-entropy and chaotic, so don't try to compress them, just retrieve them on demand. Use K to decide which V entries deserve to exist in VRAM at any given step. No model weights are modified. No retraining or distillation. It hooks into the HuggingFace cache interface and registers a custom attention function. The model has no idea it's talking to a tiered memory system. Works with any model that uses DynamicCache. Tested on Gemma 4, Qwen2.5, TinyLlama, and Phi-3.5 across MQA/GQA/MHA. There are real limitations and I'm upfront about them in the repo. Bounded prefill loses some info for dense similar-looking data. Collision disambiguation doesn't work but that's the 4-bit 2B model struggling, not the cache. Two-hop reasoning fails for the same reason. CPU RAM scales linearly (5.8GB at 1M tokens). Still actively optimizing decode speed, especially at longer contexts. The current bottleneck is CPU-to-GPU transfer for retrieved tokens, not the model itself. Plenty of room to improve here. GitHub: github.com/Babyhamsta/KIV (can be installed as a local pip package, no official pip package yet) Happy to answer questions about the architecture or results. Would love to see what happens on bigger models with more VRAM if anyone wants to try it. submitted by /u/ThyGreatOof [link] [comments]