A structured, evidence-backed page for the PromptTriage app, research studies, architecture, deployment history, Azure ML costs, tradeoffs, and remaining proof gaps.
RAG-powered prompt engineering platform with a documented research trail, modality-specific optimization, and a public demo.
PromptTriage was built around several connected studies, not a single benchmark. The public articles cover the most readable findings, while the local research logs preserve model, RAG, judge-bias, and stopped/resumed run details.
Analyzed a large corpus of production system prompts to identify recurring anti-patterns: persona bloat, contradictory long-prompt clauses, and emotional-pressure scaffolds that distracted from the actual task.
Hashnode 28K writeup, release chart, Pinecone corpus evidence.Curated publicly leaked system prompts attributed to major AI products and labs, including Anthropic, OpenAI, Google, Perplexity, Cursor, v0, Windsurf, and Gemini CLI, then used them as reference patterns instead of republishing private prompt text on the portfolio page.
Local PromptTriage repo references, training data notes, RAG corpus docs.The public writeup describes five frontier models evaluated across 12 task domains, six output formats, and three prompt lengths. The published article says each result was blindly scored by a three-judge LLM panel; local research notes also preserve earlier Study E run details and cross-model judging variants.
Hashnode Format Wars article, release charts, local RESEARCH_PROGRESS.md.Several studies exposed judge bias, especially self-preference from the original Gemini judge. Later passes used neutral Llama 4 Maverick judging or cross-model judging, so the case study records both positive results and evaluator limitations.
PromptTriage research progress log and benchmark notes.The full research writeups currently live on Hashnode. This page treats them as source evidence and connects them to the structured case-study template.
Hashnode article covering the 28K prompt corpus, production anti-patterns, and why prompt hygiene became the core product thesis.
Read on Hashnode →Hashnode article covering the 1,080-eval format study and the measurable impact of prompt structure across models and domains.
Read on Hashnode →Release assets from the PromptTriage repository are shown directly here so the case study has visible evidence instead of only outbound article links.
The app showcase is now available as a GitHub release asset. Because it is about 121 MB and marketing-heavy, this portfolio links to it deliberately instead of autoplaying or treating it as primary benchmark evidence.
The release video is useful as a product walkthrough, but the case study treats it as supporting demo evidence. The main technical proof remains the benchmark logs, Azure ML runs, research charts, and implementation source.
Open release video →Azure cost evidence now includes the EUR 178.12 credit spend screenshot plus CLI-reviewed Azure ML job metadata for the highest-signal benchmark runs.
Cost screenshot groups spend under Virtual Machines in East US 2 for the qwentrain workspace period.
MoE-only Study B benchmark, showing why naive inference became the cost lesson.
Local Azure ML logs show PyTorch, CUDA, and NVIDIA A100 80GB PCIe runtime evidence.
The cost API can break the research-period VM spend into meters, but it does not directly attach billed euros to each Azure ML run ID. For reviewer accuracy, this page shows meter-level cost plus job runtime evidence instead of pretending the portal gives clean per-job billing.
Primary Azure VM meter behind the gpu-a100 benchmark and training work.
Small supporting VM meter surfaced in the same East US 2 cost query.
qwen3_30b_a3b benchmark run on Azure ML; this is the main expensive slow-inference evidence.
Qwen3-14B run across six RAG levels for 180 prompt-generation combinations.
QLoRA fine-tuning run for the dense-vs-MoE comparison.
Some runs failed, while others were intentionally stopped and resumed with the next job at the planned continuation point.
This is the technical layer behind the case study: not just an app wrapper, but a research workflow using fine-tuning, retrieval, benchmarks, and judged evaluation.
Used to patch Qwen-family training and inference paths for faster fine-tuning and lower VRAM pressure on Azure ML GPU runs.
Fine-tuned compact adapters for Qwen variants instead of retraining full base models, keeping experiments feasible on limited credits.
Ran generation and evaluation workloads on Azure ML gpu-a100 compute, including the long MoE benchmark that exposed inference cost.
Compared six retrieval levels and found simple Pinecone retrieval more useful than heavier orchestration for this prompt-generation path.
Used judged outputs and re-judging passes to compare model, prompt format, and strategy quality while tracking judge-bias risks.
Some Azure ML jobs were stopped intentionally and resumed in follow-up jobs, so not every non-completed job represents a broken experiment.
This separates benchmark infrastructure from app hosting so reviewers do not confuse the research environment with the product runtime.
Current-build view based on the local PromptTriage source: Supabase Auth and Stripe sit in the app layer, FastAPI handles retrieval, Pinecone stores prompt vectors, and GitHub Actions deploys containers to Azure.
The table follows the case-study template and separates risks, implemented controls, and current evidence sources.
| Risk | Control | Evidence |
|---|---|---|
| Unauthenticated use of paid or rate-limited workflows | Supabase Auth sessions are checked by server routes before analyze, refine, and system-prompt generation. | Local source: Supabase middleware and API route auth checks. |
| Subscription state drift between payment and app access | Stripe checkout, billing portal, and webhooks write subscription state into Supabase. | Local source: Stripe webhook and subscription routes. |
| Secret leakage from deployment pipeline | Azure OIDC is used for GitHub Actions deployment, with runtime secrets stored outside source code. | Local source: deploy-frontend.yml and deploy-backend.yml. |
| Vector database or model provider exposure | Pinecone and provider keys are server-side environment variables; the browser only talks to application routes. | Local source: backend RAG service and environment examples. |
The implementation evidence now points to Azure Container Apps rather than the older deployment notes.
These are the items from the professional case-study template that still need more source material before the PromptTriage page becomes fully audit-ready.
Job-level Azure ML cost allocation that maps run IDs to billed spend. Current evidence itemizes the EUR 178.12 VM meters, but Azure Cost Management does not directly attach every euro to each ML job.
Public-safe screenshots or summaries for monitoring, alerts, rollback, incident response, and restore paths.
Screenshots or redacted config evidence for Supabase Auth, Stripe webhook verification, secrets, and deployment identity.
Visual evidence for Azure Container Apps, ACR, Google Cloud Run history, GitHub Actions runs, image scanning, and rollback.
Exportable SVG is now included; optional PNG, Excalidraw, or Draw.io source can be added if you want a diagram asset that is easier to reuse in decks.
A compact public table that maps each study to sample size, model set, evaluation method, and final takeaway.
PromptTriage is a RAG-powered prompt engineering platform built to turn rough ideas into production-ready prompts across text, image, video, and system-prompt workflows. The current build combines a Next.js interface, Supabase Auth, Stripe subscription flows, FastAPI retrieval services, Pinecone vector search, modality-specific metaprompts, and Azure Container Apps deployment. The strongest evidence is not only the app surface, but the documented research process behind retrieval strategy, prompt format, model selection, evaluation bias, and cost tradeoffs.
Most prompt tools behave like thin wrappers over a model API. The project needed to show whether structured prompt generation, retrieval context, and modality-specific templates could produce better, more reliable prompts than generic chat UX. The user problem is practical: builders need reusable prompts that include constraints, output format, assumptions, and evaluation criteria without manually writing every section from scratch.
The project was developed as a solo engineering effort with limited budget, evolving model availability, provider API instability, GPU quota constraints, and a need to publish evidence without exposing secrets or private datasets. Most benchmark and training work ran on Azure ML compute clusters, while the app moved through Azure Container Apps and Google Cloud Run deployment phases. Research notes show cost-sensitive experimentation across fine-tuned Qwen variants, RAG strategies, and LLM-as-judge workflows.
Functional requirements included prompt analysis, dynamic clarification questions, modality routing, retrieval from a prompt corpus, final prompt synthesis, one-click iteration, paid-plan capability checks, and optional current-documentation lookup through MCP tooling. Non-functional requirements included secure authentication, environment-based secrets, responsive UI, reproducible prompt versions, structured logging, and a containerized path from GitHub to Azure runtime. Research requirements included Unsloth-based QLoRA experiments, Azure ML benchmark jobs, and repeatable evaluation outputs.
The architecture separates the user-facing workflow from retrieval and prompt-engineering services. The UI collects intent and modality, Next.js API routes enforce Supabase session and subscription context, the backend retrieves relevant examples from Pinecone, and modality-specific metaprompts synthesize the final result. Azure Container Apps hosts the containerized frontend and backend services through images built and pushed by GitHub Actions.
The current security model centers on Supabase Auth, server-side subscription checks, Stripe webhook verification, environment-based secrets, Azure OIDC deployment, and no publication of private provider keys or vector indexes. The risk/control table below is intentionally evidence-oriented so reviewers can see which controls are implemented and which supporting artifacts still need screenshots.
The local workflows document a frontend and backend Docker deployment path to Azure Container Apps through Azure Container Registry. Historically, PromptTriage also ran on Google Cloud Run. The Azure app runtime and ACR were later accidentally removed during CLI cleanup, which is now part of the operational lessons learned rather than hidden from the case study.
Operational evidence includes API logging, health routes, error feedback UX, dataset ingestion scripts, research progress notes, and failure tracking around model/provider behavior. The next operational step is to attach Azure Container Apps logs, alert thresholds, incident response notes, rollback proof, and cost-monitoring screenshots to make the operations section audit-ready.
The Azure cost screenshot shows EUR 178.12 of credits spent for the PromptTriage research period, grouped under Virtual Machines in East US 2 for the qwentrain workspace. Azure ML job metadata shows the longest completed MoE benchmark ran for 38.19 hours on the gpu-a100 compute target, while Study A ran for 5.35 hours. The main cost lesson is that benchmark design, retry behavior, and inference implementation can matter more than raw model size.
Documented results include a 28K+ prompt evidence base, Pinecone retrieval, multiple completed studies, qwen3_14b selected as a production candidate in the research notes, Unsloth/QLoRA model experiments, and a finding that naive RAG performed better than more complex RAG variants for this pipeline. Some downstream benchmark results were nuanced, so the case study should preserve both positive findings and null results instead of overstating uniform gains.
The main tradeoff was simplicity versus orchestration complexity. Naive RAG was easier to operate and outperformed more agentic retrieval variants in the documented study, while complex corrective or judge-based RAG added noise. The project also trades broad provider coverage for a more controlled, research-backed prompt workflow.
The research logs are valuable because they document failures and controlled interruptions: judge bias, provider/model availability issues, expensive slow MoE inference, failed Azure ML runs, planned stop/resume benchmark jobs, overfitting on small datasets, prompt strategies that did not move downstream scores, and accidental cloud-resource teardown. These failures make the project stronger as engineering evidence because they show measurement discipline rather than only demo polish.
Next improvements should include a formal runbook, security model page, threat model, Azure cost report, architecture diagram export stored in the repo, API rate limiting, clearer public benchmark reports, and a more complete deployment checklist. Those additions align directly with the documentation standard from the career execution pack.
The public demo has historically been hosted at prompttriage.kaelux.dev and the repository is available on GitHub. The strongest supporting evidence is the README, backend research notes, Azure ML logs, benchmark output JSON, deployment workflows, security policy, release charts, and the marketing-oriented app showcase video.
I built PromptTriage because prompt quality is usually treated as subjective, but production AI systems need measurable prompt structure, retrieval context, and evaluation discipline. The architecture uses a Next.js interface, Supabase Auth, Stripe billing, API orchestration, FastAPI retrieval, Pinecone vectors, Azure Container Apps, and modality-specific metaprompts. The most important lesson was that more complex RAG is not automatically better: in this project, simple retrieval produced stronger documented results than agentic retrieval traces.
Designed a RAG-powered prompt engineering platform with modality-specific optimization, Pinecone retrieval, and documented evaluation studies. Built and evaluated prompt-generation workflows across model selection, retrieval strategy, judge bias, and format sensitivity. Documented cost, failure modes, and tradeoffs to make the project inspectable by engineering reviewers.
The main case-studies page separates PromptTriage from Kaelux.dev, ViperMesh, and n8n Automation Atlas so each project can grow into its own structured page.