Product Development Path

This is a highly efficient development environment for this project. Developing the core logic and models within WSL2 is the best path because it provides a full Linux environment, which is the native home for most open-source LLM tools, while still leveraging your Windows 11 hardware (especially the GPU).

Here is the recommended development path, broken down into 4 clear stages.

Stage 1: WSL Setup and Tooling (The Foundation)¶

1. Configure WSL2 with Ubuntu¶

Install WSL2: Ensure you are running WSL version 2 (e.g., Ubuntu 20.04 or 22.04).
Enable GPU Support (Crucial): If your machine has an NVIDIA or AMD GPU, you must configure it for use within WSL2. This is essential for fast model loading and testing.
- NVIDIA: Install the latest NVIDIA drivers on Windows and the CUDA Toolkit within the WSL environment.

2. Set up the Python Development Environment¶

Install Python: Use a version like Python 3.10 or 3.11 within your WSL distribution.
Virtual Environment: Always use a virtual environment (venv or conda) to manage dependencies.

Install Core Libraries:

# Inside WSL terminal
sudo apt update
sudo apt install python3-pip cmake git
python3 -m venv llm_museum_env
source llm_museum_env/bin/activate
pip install torch torchvision numpy

3. Build the Inference Engine (`llama.cpp`)¶

Clone and Build: llama.cpp will be your primary engine for running the local VLM (Qwen).

git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
make
# If you have CUDA configured:
# LLAMA_CUBLAS=1 make

Result: This gives you the C++ binaries necessary for model conversion and, crucially, the llama-cpp-python bindings for easy prototyping.

Stage 2: Core VLM Prototyping (The Intelligence)¶

This stage involves getting the vision and language model working on your desktop.

1. Model Acquisition and Quantization¶

Download the VLM: Get the recommended model, Qwen2.5-VL-7B-Instruct, from Hugging Face. Download the PyTorch or other base weights to your WSL file system.
Convert to GGUF: Use the Python tools within the llama.cpp repository to convert the model weights into the GGUF format. This highly-quantized format is what will run efficiently on the mobile device's CPU/NPU.
- Example: Running the conversion script (typically a two-step process: convert.py then quantize).

2. Desktop Multimodal Testing¶

Test Script: Write a Python script (using llama-cpp-python bindings) that loads the new GGUF file.
Test Flow:
1. Load a test museum image (e.g., from your WSL files).
2. Define a prompt (e.g., "Analyze this image and describe the artist's technique.").
3. Pass the image and prompt to the model and measure the time-to-first-token (TTFT) and total generation time.
Goal: Confirm the VLM runs correctly and quickly on your WSL machine, proving the GGUF file is valid for the next step (mobile).

Stage 3: Full-Stack Integration (The Prototype Chain)¶

Now, you integrate the input (STT) and output (TTS) to create the full desktop prototype.

1. Implement Local Speech-to-Text (Vosk)¶

Install Vosk:
```
pip install vosk
```
Download Model: Download a small, local Vosk language model (e.g., vosk-model-en-us-0.22) to your project directory.
Test Script: Write a script that uses a microphone (accessible through Windows/WSL audio forwarding) or a pre-recorded audio file to generate the text prompt.

2. Implement Local Text-to-Speech (TTS)¶

Recommended TTS: Use a local, open-source library that can run on CPU, such as MeloTTS or a similar lightweight option from Hugging Face.
```
# Example for a simple CPU-friendly TTS
pip install transformers accelerate soundfile
```
Test Script: Use the script to convert the VLM's generated text response into an audio file (e.g., .wav or .mp3).

3. Final Desktop Chaining¶

Connect the Components: Create a main Python script that runs the full user flow, using the components you just tested:
1. Audio Input (Microphone) \(\rightarrow\) Vosk \(\rightarrow\) Text Prompt.
2. Image + Text Prompt \(\rightarrow\) Qwen VLM (GGUF) \(\rightarrow\) AI Text Response.
3. AI Text Response \(\rightarrow\) MeloTTS \(\rightarrow\) Audio Output (Playback).
Goal: Have a fully functional, end-to-end, local prototype running on your Windows/WSL machine.

Stage 4: Mobile App Transition (Deployment Path)¶

This is the final step, taking the validated logic and moving it to a mobile framework.

1. Choose a Mobile Development Framework¶

Based on your successful WSL prototyping, the most efficient mobile framework will be:

Framework	Reason
React Native (Recommended)	Fastest to prototype, cross-platform (iOS/Android).
Native Kotlin/Swift	Highest performance ceiling, but requires more platform-specific code.

2. Implement Mobile Inference Bindings¶

You need the C++ logic from llama.cpp to be callable from your mobile app's JavaScript/Kotlin/Swift code.

For React Native: Use a pre-built community library like llama.rn or react-native-llama-cpp. These libraries already handle the complex compilation of llama.cpp into an Android AAR and iOS Framework.
For Native: Manually compile llama.cpp (and the VLM/TTS components if they are C/C++ based) as a library for Android JNI and/or an iOS Framework.

3. Deploy the GGUF Models¶

Package the Assets: The single, large GGUF file for Qwen, the Vosk model files, and the TTS model files are packaged directly into the app's assets folder.
Final App Flow: The mobile app calls the local C++/llama.cpp bindings, providing the image and the prompt, ensuring the entire conversation remains fast and offline.

That is an absolutely critical and valid concern. You have hit upon the biggest limitation of using a foundational LLM (especially a small, local one) for a highly specialized domain like museum studies or archaeology.

Here is a breakdown of the limitation and the necessary technical solution to overcome it.

1. The Limitation: Generalist vs. Specialist Knowledge¶

A model like Qwen2.5-VL-7B-Instruct is a generalist VLM.

What it knows (General Knowledge): It knows about famous pieces (e.g., the Mona Lisa, the Rosetta Stone) and can describe general artistic styles (Baroque, Impressionism) because this information is ubiquitous on the internet (its training data).
What it **doesn't know (Specific Knowledge):** It will almost certainly not know the specific exhibit number, the provenance history, or the detailed description written on the placard for Exhibit 42B in the corner of the Greek Wing of a specific museum. That information is too niche and was not present in the model's massive, but still finite, general training data.

If you simply ask the base LLM, it will likely either hallucinate a plausible but incorrect answer, or say "I don't know."

2. The Solution: Local Retrieval-Augmented Generation (RAG)¶

To solve this, you must augment the LLM's general intelligence with a specialized, local knowledge base. This architecture is called Retrieval-Augmented Generation (RAG).

RAG in your local app works like this:

Step A: The Local Knowledge Base (The Curator's Brain)¶

Data Acquisition: You need the museum/archaeological site data. This is the hardest part and must be done upfront.
- Sources: Museum website descriptions, exhibit PDFs, digitized catalogs, label text, and peer-reviewed articles.
Indexing (Vectorization): The plain text data is broken down into small, meaningful chunks. These chunks are converted into vector embeddings (numerical representations of meaning) using a small, local embedding model (e.g., a bge-small or similar model).
Local Database: These vectors are stored in a Local Vector Database on the phone.
- Recommended Local DB: DuckDB or a simple flat file index using Faiss or Hnswlib which are optimized for on-device vector storage and retrieval.

Step B: The Retrieval Flow¶

When the user takes a picture and asks a question, the process changes:

VLM First Pass (Visual ID): The image is fed to the Qwen VLM with a prompt like: "Analyze this image and output only the name and a 5-word description of the object. Do not elaborate."
- Output: "Bronze Roman bust of Emperor Claudius."
Vector Search (Retrieval): The VLM's output (or a combination of the image embedding and the user's question) is used to query the Local Vector Database (Faiss/Hnswlib).
- Retrieval: The database finds the top 3 most relevant chunks of text from your specialized museum data (e.g., the entry from the museum's catalog on "Emperor Claudius Bust").
VLM Second Pass (Generation): These retrieved text chunks are added to the user's original question to create a new, enriched prompt for the VLM.
- Final Prompt: "Based on the following context: [Retrieved Text Chunks], answer the user's question: 'What is the provenance and display history of this bust?'"
Final Output: The VLM now generates a detailed, accurate, and non-hallucinated answer, which is sent to the TTS model.

RAG is the ONLY viable path for this application.¶

Implementing RAG ensures:

Accuracy: The VLM is forced to ground its response in the authoritative museum data you provided.
Up-to-Date Knowledge: You can update the RAG database (the vectors) anytime without having to retrain the entire large LLM.
Small Model Efficiency: A small, local LLM can handle the general logic and conversational flow while outsourcing the heavy knowledge burden to the dedicated RAG system.

This is a complex, high-value project that touches on several cutting-edge areas (local multimodal AI, RAG, and cross-platform native bindings).

Based on the highly specialized stack you've chosen (local VLM, RAG, STT, TTS, and mobile deployment), the estimated time to reach a Stable Beta on a single platform (iOS or Android) is approximately 10 to 19 Weeks (2.5 to 5 months), assuming an experienced, full-time developer or a small, focused team.

The timeline is heavily weighted toward the Data Acquisition (RAG) and Mobile Integration phases.

Detailed Project Timeline Breakdown¶

The project should be broken into four sequential phases. Time estimates are for a dedicated, experienced developer.

Phase 1: Foundation & Core VLM Prototyping (2-3 Weeks)¶

Task	Estimated Time	Rationale
WSL/Dev Environment Setup	3-5 days	Configuring WSL2, ensuring CUDA/GPU passthrough, building `llama.cpp` and necessary Python wrappers.
VLM Acquisition & Quantization	4-6 days	Downloading Qwen2.5-VL-7B, converting to GGUF, and testing basic inference on WSL. Fine-tuning model parameters for speed/memory.
Input/Output Model Testing	3-5 days	Integrating and testing Vosk (STT) and a lightweight TTS model like MeloTTS separately.
Phase 1 Total	2 Weeks	Focus on speed, efficiency, and stable GGUF/C++ inference on the desktop.

Phase 2: Knowledge Base and Local RAG Pipeline (4-7 Weeks)¶

This is the most critical and highest-variance phase, depending entirely on the source data.

Task	Estimated Time	Rationale
Data Acquisition & Cleaning	2-4 Weeks	Sourcing museum and archaeological data (PDFs, websites, catalog exports). This involves cleaning messy, unstructured data into a usable format. The major variable.
Embedding Model Setup	4-6 days	Implementing a local embedding model (e.g., bge-small) and creating the scripts for chunking the text data.
Local Vector DB Implementation	5-7 days	Choosing and setting up the on-device vector database (e.g., Faiss or Hnswlib) and generating the first full index of museum data.
RAG Logic Implementation	5-7 days	Writing the core RAG script: VLM Visual ID \(\rightarrow\) Query DB \(\rightarrow\) VLM Generation. Debugging retrieval effectiveness (the quality of the answer).
Phase 2 Total	4-7 Weeks	Focus on getting highly accurate answers based on the specialized knowledge.

Phase 3: Full Desktop Prototype Integration (3-4 Weeks)¶

Task	Estimated Time	Rationale
End-to-End Chaining	5-7 days	Connecting STT \(\rightarrow\) VLM/RAG \(\rightarrow\) TTS into one fluid Python script on the WSL environment.
Conversation Management	5-7 days	Implementing history and context handling so the AI can answer follow-up questions ("Tell me more about it").
Latency & Performance Tuning	5-7 days	Iterating on model settings (quantization, threads) to ensure a response time under 5-8 seconds—the maximum acceptable latency for a conversational app.
Phase 3 Total	3-4 Weeks	Focus on a stable, fast, conversational prototype that mimics the final app experience.

Phase 4: Mobile App Transition & Deployment (4-8 Weeks)¶

This is the most technically challenging phase, requiring integration of C++ logic into a mobile environment.

Task	Estimated Time	Rationale
Mobile Framework Setup	3-5 days	Setting up the React Native (or native) project and basic UI structure.
C++ / Native Bindings	2-4 Weeks	Building the C++ wrapper for `llama.cpp` and the local RAG/Vosk/TTS logic into a reusable library (e.g., an iOS Framework or Android AAR) that the UI can call. This is highly specialized work.
UI/UX Implementation	1-2 Weeks	Designing the Camera, Speak, and Display interfaces to be intuitive for museum visitors.
Deployment & Beta Testing	1-2 Weeks	Final memory, battery, and speed testing on target mobile devices. Debugging platform-specific issues (e.g., memory limits on iOS/Android).
Phase 4 Total	4-8 Weeks	Focus on reliable, multi-platform deployment of the heavy local models.

Key Risk Factors & Variables¶

Data Availability (Phase 2): If your target museum's data is clean, well-structured, and easily exportable, Phase 2 could be shorter. If it requires extensive manual scraping and cleanup, it will take longer.
Mobile Bindings (Phase 4): This step is often the project killer for complex local AI apps. If you must build the C++ bindings from scratch rather than using existing community tools (like llama.rn), the timeline will shift to the higher end of the estimate.
Performance (Phase 3): Achieving the required low latency (real-time response) on older or lower-end phones might require more time for aggressive model quantization and low-level code optimization.

This is a great approach. Parallelizing development is essential for a project of this complexity. You can break the project into three distinct, loosely coupled Vertical Pillars (the core function) and then define the Interfaces (APIs) that allow them to communicate.

By defining clear interfaces, different parts can be developed simultaneously by different people or focused work sprints.

1. The Three Parallel Vertical Pillars¶

You can divide the development into three major areas that require different skill sets:

Pillar	Focus	Core Skill Set	Time/Dependency
A. Front-End / UX (Mobile App)	User Interface, Camera/Mic Handling, Final App Assembly.	React Native, Mobile UI/UX.	Least Dependent: Needs only the final C++ bindings to plug in.
B. Data & Knowledge (RAG Pipeline)	Data Acquisition, Vectorization, Retrieval Logic.	Data Engineering, Python, Vector Databases.	Most Independent: Can be developed entirely in isolation within WSL.
C. Core AI Engine (Model & Runtime)	Model loading, `llama.cpp` integration, STT/TTS calls.	C++, Model Optimization (GGUF), Native Bindings (JNI/Swift).	High Dependency: Needs the RAG output format and must expose clean I/O to the Front-End.

2. Suggested Interfacing (APIs) for Parallel Development¶

The interfaces must define the exact input and output for each function call across the pillars. Everything should be standardized in simple, reliable formats (JSON objects and file paths/buffers).

Interface 1: The Core Query API (Pillar A \(\leftrightarrow\) Pillar C)¶

This is the main interaction point between the mobile UI and the local AI processing engine.

Attribute	Input (from Mobile App)	Output (to Mobile App)
Function Name	`StartConversation()`
Image Input	Base64-encoded image data or a local file URI.
Text Input	Transcribed user question (from local STT).
Conversation ID	A unique ID for session continuity (for follow-ups).
Output Type		A JSON object: `{ "text_response": "...", "audio_buffer": [Raw Audio Data] }`
Status Type		An Enumeration (e.g., `Processing`, `Ready`, `Error`).

Goal: The Front-End (Pillar A) simply calls this function with the image/text and receives a ready-to-play audio buffer, without knowing anything about RAG or the model type.

Interface 2: The RAG Knowledge API (Pillar C \(\leftrightarrow\) Pillar B)¶

This is how the Core AI Engine gets the specialized context it needs from the local knowledge base.

Attribute	Input (from Core AI Engine)	Output (to Core AI Engine)
Function Name	`RetrieveContext()`
Query Text	VLM's initial visual identification + user question (e.g., "Roman Bust" and "tell me its history").
Knowledge Index Path	Path to the local Vector Database file (e.g., `/app_data/museum_index.faiss`).
Output Type		A text string of concatenated, relevant chunks: `"Context: [chunk 1...], [chunk 2...], [chunk 3...]"`.

Goal: The RAG team (Pillar B) focuses only on making RetrieveContext() return the most accurate, concise text possible, entirely separate from the VLM/Mobile code.

Interface 3: Local TTS/STT Wrappers (Internal to Pillar C)¶

While internal to the AI Engine, defining these early ensures portability.

Function Name	Input	Output
`TranscribeAudio(audio_path)`	File path to user's recorded audio.	Clean text string.
`TextToAudio(text_string)`	AI-generated text response.	Raw audio buffer or local audio file path.

3. Parallel Development Strategy¶

Using these interfaces, you can break the project into parallel tracks:

Track	Work Focus	Time Frame
Track 1: Data Acquisition & RAG (Pillar B)	Full Time (Weeks 1-7): Scrape, clean, chunk, and vectorize all museum data. Test the RAG effectiveness using simple Python scripts on WSL. This is the critical path for accuracy.
Track 2: Mobile UI & UX (Pillar A)	Full Time (Weeks 1-4): Build the full mobile app UI, including camera and microphone handlers, but for testing, have the button call a mocked function that returns a hardcoded text string and a pre-recorded audio file. (Mock Interface 1)
Track 3: Core AI Engine (Pillar C)	Full Time (Weeks 1-8): Work on the C++ bindings. First, get the VLM running fast. Then, integrate the mocked RAG (e.g., hardcoded context). Finally, integrate the live RAG via Interface 2 and the STT/TTS wrappers.

By the end of Week 4-5, you should have a functional, user-friendly UI Mock (Pillar A) and the RAG Knowledge Base (Pillar B) complete. The remaining time is focused on the difficult task of integrating the AI/C++ code (Pillar C) into the mobile environment (Pillar A).

Hardware Requirements¶

This is a complex, high-value project that touches on several cutting-edge areas (local multimodal AI, RAG, and cross-platform native bindings).

Based on the highly specialized stack you've chosen (local VLM, RAG, STT, TTS, and mobile deployment), the estimated time to reach a Stable Beta on a single platform (iOS or Android) is approximately 10 to 19 Weeks (2.5 to 5 months), assuming an experienced, full-time developer or a small, focused team.

The timeline is heavily weighted toward the Data Acquisition (RAG) and Mobile Integration phases.

Detailed Project Timeline Breakdown¶

The project should be broken into four sequential phases. Time estimates are for a dedicated, experienced developer.

Phase 1: Foundation & Core VLM Prototyping (2-3 Weeks)¶

Task	Estimated Time	Rationale
WSL/Dev Environment Setup	3-5 days	Configuring WSL2, ensuring CUDA/GPU passthrough, building `llama.cpp` and necessary Python wrappers.
VLM Acquisition & Quantization	4-6 days	Downloading Qwen2.5-VL-7B, converting to GGUF, and testing basic inference on WSL. Fine-tuning model parameters for speed/memory.
Input/Output Model Testing	3-5 days	Integrating and testing Vosk (STT) and a lightweight TTS model like MeloTTS separately.
Phase 1 Total	2 Weeks	Focus on speed, efficiency, and stable GGUF/C++ inference on the desktop.

Phase 2: Knowledge Base and Local RAG Pipeline (4-7 Weeks)¶

This is the most critical and highest-variance phase, depending entirely on the source data.

Task	Estimated Time	Rationale
Data Acquisition & Cleaning	2-4 Weeks	Sourcing museum and archaeological data (PDFs, websites, catalog exports). This involves cleaning messy, unstructured data into a usable format. The major variable.
Embedding Model Setup	4-6 days	Implementing a local embedding model (e.g., bge-small) and creating the scripts for chunking the text data.
Local Vector DB Implementation	5-7 days	Choosing and setting up the on-device vector database (e.g., Faiss or Hnswlib) and generating the first full index of museum data.
RAG Logic Implementation	5-7 days	Writing the core RAG script: VLM Visual ID \(\rightarrow\) Query DB \(\rightarrow\) VLM Generation. Debugging retrieval effectiveness (the quality of the answer).
Phase 2 Total	4-7 Weeks	Focus on getting highly accurate answers based on the specialized knowledge.

Phase 3: Full Desktop Prototype Integration (3-4 Weeks)¶

Task	Estimated Time	Rationale
End-to-End Chaining	5-7 days	Connecting STT \(\rightarrow\) VLM/RAG \(\rightarrow\) TTS into one fluid Python script on the WSL environment.
Conversation Management	5-7 days	Implementing history and context handling so the AI can answer follow-up questions ("Tell me more about it").
Latency & Performance Tuning	5-7 days	Iterating on model settings (quantization, threads) to ensure a response time under 5-8 seconds—the maximum acceptable latency for a conversational app.
Phase 3 Total	3-4 Weeks	Focus on a stable, fast, conversational prototype that mimics the final app experience.

Phase 4: Mobile App Transition & Deployment (4-8 Weeks)¶

This is the most technically challenging phase, requiring integration of C++ logic into a mobile environment.

Task	Estimated Time	Rationale
Mobile Framework Setup	3-5 days	Setting up the React Native (or native) project and basic UI structure.
C++ / Native Bindings	2-4 Weeks	Building the C++ wrapper for `llama.cpp` and the local RAG/Vosk/TTS logic into a reusable library (e.g., an iOS Framework or Android AAR) that the UI can call. This is highly specialized work.
UI/UX Implementation	1-2 Weeks	Designing the Camera, Speak, and Display interfaces to be intuitive for museum visitors.
Deployment & Beta Testing	1-2 Weeks	Final memory, battery, and speed testing on target mobile devices. Debugging platform-specific issues (e.g., memory limits on iOS/Android).
Phase 4 Total	4-8 Weeks	Focus on reliable, multi-platform deployment of the heavy local models.

Key Risk Factors & Variables¶

Data Availability (Phase 2): If your target museum's data is clean, well-structured, and easily exportable, Phase 2 could be shorter. If it requires extensive manual scraping and cleanup, it will take longer.
Mobile Bindings (Phase 4): This step is often the project killer for complex local AI apps. If you must build the C++ bindings from scratch rather than using existing community tools (like llama.rn), the timeline will shift to the higher end of the estimate.
Performance (Phase 3): Achieving the required low latency (real-time response) on older or lower-end phones might require more time for aggressive model quantization and low-level code optimization.

Product Development Path

Stage 1: WSL Setup and Tooling (The Foundation)¶

1. Configure WSL2 with Ubuntu¶

2. Set up the Python Development Environment¶

3. Build the Inference Engine (llama.cpp)¶

Stage 2: Core VLM Prototyping (The Intelligence)¶

1. Model Acquisition and Quantization¶

2. Desktop Multimodal Testing¶

Stage 3: Full-Stack Integration (The Prototype Chain)¶

1. Implement Local Speech-to-Text (Vosk)¶

2. Implement Local Text-to-Speech (TTS)¶

3. Final Desktop Chaining¶

Stage 4: Mobile App Transition (Deployment Path)¶

1. Choose a Mobile Development Framework¶

2. Implement Mobile Inference Bindings¶

3. Deploy the GGUF Models¶

1. The Limitation: Generalist vs. Specialist Knowledge¶

2. The Solution: Local Retrieval-Augmented Generation (RAG)¶

Step A: The Local Knowledge Base (The Curator's Brain)¶

Step B: The Retrieval Flow¶

RAG is the ONLY viable path for this application.¶

Detailed Project Timeline Breakdown¶

Phase 1: Foundation & Core VLM Prototyping (2-3 Weeks)¶

Phase 2: Knowledge Base and Local RAG Pipeline (4-7 Weeks)¶

Phase 3: Full Desktop Prototype Integration (3-4 Weeks)¶

Phase 4: Mobile App Transition & Deployment (4-8 Weeks)¶

Key Risk Factors & Variables¶

1. The Three Parallel Vertical Pillars¶

2. Suggested Interfacing (APIs) for Parallel Development¶

Interface 1: The Core Query API (Pillar A \(\leftrightarrow\) Pillar C)¶

Interface 2: The RAG Knowledge API (Pillar C \(\leftrightarrow\) Pillar B)¶

Interface 3: Local TTS/STT Wrappers (Internal to Pillar C)¶

3. Parallel Development Strategy¶

Hardware Requirements¶

Detailed Project Timeline Breakdown¶

Phase 1: Foundation & Core VLM Prototyping (2-3 Weeks)¶

Phase 2: Knowledge Base and Local RAG Pipeline (4-7 Weeks)¶

Phase 3: Full Desktop Prototype Integration (3-4 Weeks)¶

Phase 4: Mobile App Transition & Deployment (4-8 Weeks)¶

Key Risk Factors & Variables¶

3. Build the Inference Engine (`llama.cpp`)¶