[ { "path": ".gitignore", "content": "# Python cache\n__pycache__/\n*.py[cod]\n*.so\n\n# Virtual environments\n.venv/\nenv/\n.log/\n# Secrets & configs\n.env\n*.env\n*.yml\n*.sh\n*.bat\ndevice_ips.txt\nadb.txt\n\n# Logs & temporary files\n*.log\n*.tmp\n*.swp\n\n# System files\n.DS_Store\nThumbs.db\nehthumbs.db\nIcon?\n\n# Data & storage\nDATA/\n*.pkl\n*.db\n*.sqlite3\n*.csv\n*.json\n**/VECTORSTORES/\n*.faiss\n\n# Backups & test outputs\nBackup/\nTest/\n\n# IDEs\n.vscode/\n.idea/\n\n# Media (test/generated files)\n*.mp3\n*.wav\n*.mp4\n*.m4a\n*.png\n*.jpg\n*.jpeg\n*.gif\n\n# Packaging\nbuild/\ndist/\n*.egg-info/\n\n# Jupyter\n.ipynb_checkpoints/\n\n# Caches\n.pytest_cache/\n.mypy_cache/\n.cache/\n" }, { "path": "readme.md", "content": "# šŸš€ JARVIS 2.0\n\n---\n## J.A.R.V.I.S. 2.0 ā€“ Judgment Augmented Reasoning for Virtual Intelligent Systems\n# šŸ¤– Jarvis AI Assistant\n\nWelcome to the **Jarvis AI Assistant** project! šŸŽ™ļø This AI-powered assistant can perform various tasks such as **providing weather reports šŸŒ¦ļø, summarizing news šŸ“°, sending emails šŸ“§** , **CAG** , and more, all through **voice commands**. Below, you'll find detailed instructions on how to set up, use, and interact with this assistant. šŸŽ§\n\n---\n\n## šŸŒŸ Features\n\nāœ… **Voice Activation**: activate listening mode. šŸŽ¤\\\nāœ… **Speech Recognition**: Recognizes and processes user commands via speech input. šŸ—£ļø\\\nāœ… **AI Responses**: Provides responses using AI-generated **text-to-speech** output. šŸŽ¶\\\nāœ… **Task Execution**: Handles multiple tasks, including:\n\n- šŸ“§ **Sending emails**\n- šŸŒ¦ļø **Summarizing weather reports**\n- šŸ“Š **Data Analysis using csv***\n- šŸ§‘šŸ»ā€šŸ’» **Pesonalize chat**\n- šŸ“° **Reading news headlines**\n- šŸ–¼ļø **Image generation**\n- šŸ¦ **Database functions**\n- šŸ“± **Phone call automation using ADB**\n- šŸ¤– **AI-based task execution**\n- šŸ“” **Automate websites & applications**\n- šŸžļø **Image processing Using gemini** \n\n **Image Source:**\n ***Upload***\n ***URL***\n ***Camera***\n\n **Select Action:**\n ***Basic Detection***\n ***Object Detection***\n ***Segmentation***\n ***Resize***\n- šŸ§  **Retrieval-Augmented Generation (RAG) for knowledge-based interactions on various topics**\n- āœ… **Timeout Handling**: Automatically deactivates listening mode after **5 minutes** of inactivity. ā³\n- āœ… **Automatic Input Processing**: If no \"stop\" command is detected within **60 seconds**, input is finalized and sent to the AI model for processing. āš™ļø\n- āœ… **Multiple Function Calls**: Call **multiple functions simultaneously**, even if their inputs and outputs are unrelated. šŸ”„\n\n---\n\n## šŸ“Œ Prerequisites\n\nBefore running the project, ensure you have the following installed:\n\nāœ… **Python 3.9 or later** šŸ\\\nāœ… Required libraries (listed in `requirements.txt`) šŸ“œ\n\n### šŸ› ļø Configuration\n\n1. **Create a ************`.env`************ file** in the root directory of the project.\n\n2. **Add your API keys and other configuration variables** to the `.env` file:\n\n```dotenv\n author_name=\"ganeshnikhil124@gmail.com\"\n weather_link=\"https://rapidapi.com/weatherapi/api/weatherapi-com\"\n news_link=\"https://newsapi.org\"\n name=\"ganeshnikhil\"\n Rag_model=\"granite3.1-dense:2b\"\n Chat_model=\"granite3.1-dense:2b\"\n Function_call_model=\"gemma3:4b\"\n Text_to_info_model=\"gemma3:4b\"\n Image_to_text=\"llava:7b\"\n Embedding_model=\"nomic-embed-text\"\n genai_key=\"\"\n Sender_email=\"ganeshnikhil124@gmail.com\"\n Receiver_email=\"\"\n Password_email=\"\"\n Weather_api=\"\"\n News_api=\"\"\n Country=\"in\"\n DEVICE_IP=\"\"\n CSV_PATH=\"./DATA/business-employment-data-dec-2024-quarter.csv\"\n UI=\"on\"\n Yt_path=\"./DATA/youtube_video/\"\n```\n2 . Install system requriements \n\n```install\nbash ./intialize.sh\n\n```\n\n3. **Setup API Keys & Passwords** :\n\n - [šŸŒ©ļø WEATHER API](https://rapidapi.com/weatherapi/api/weatherapi-com) - Get weather data.\n - [šŸ“° NEWS API](https://newsapi.org) - Fetch latest news headlines.\n - [šŸ“§ GMAIL PASSWORD](https://myaccount.google.com/apppasswords) - Generate an app password for sending emails.\n - [šŸ§  OLLAMA](https://ollama.com) - Download models from Ollama (manual steup) .\n **install Models from ollama**\n ```\n ollama run gemma3:4b\n ollama run granite3.1-dense:2b\n ollama pull nomic-embed-text\n ```\n - [portaudio](https://files.portaudio.com/download.html) - download portaudio to work with sound.\n - [šŸ”® GEMINI AI](https://aistudio.google.com/apikey) - API access for function execution.\n\n## Model Details\n # Gemma for intellignet routing image and simple question answers.\n```\n Model\n architecture gemma3 \n parameters 4.3B \n context length 8192 \n embedding length 2560 \n quantization Q4_K_M \n\n Parameters\n stop \"\" \n temperature 0.1 \n\n License\n Gemma Terms of Use \n Last modified: February 21, 2024\n```\n # grantie dense has large context window ,for rag and chat.\n```\n Model\n architecture granite \n parameters 2.5B \n context length 131072 \n embedding length 2048 \n quantization Q4_K_M \n\n System\n Knowledge Cutoff Date: April 2024. \n You are Granite, developed by IBM. \n\n License\n Apache License \n Version 2.0, January 2004\n```\n# gemini free teir for as fallback mechanism . (only for tool calling)\n```\ngemini-2.0-flash\n Audio, images, videos, and text\tText, images (experimental), and audio (coming soon)\tNext generation features, speed, thinking, realtime streaming, and multimodal generation\ngemini-2.0-flash-lite\n Audio, images, videos, and text\tText\tA Gemini 2.0 Flash model optimized for cost efficiency and low latency\ngemini-2.0-pro-exp-02-05\n Audio, images, videos, and text\tText\tOur most powerful Gemini 2.0 model\ngemini-1.5-flash\n Audio, images, videos, and text\tText\tFast and versatile performance across a diverse variety of tasks\n```\n![JARVIS Screenshot](image.png)\n---------------------------------------------------------------------------------------------\n![alt text](dig.png)\n## šŸ’» Installation\n\n### 1ļøāƒ£ **Clone the Repository**\n\n```bash\n git clone https://github.com/ganeshnikhil/J.A.R.V.I.S.2.0.git\n cd J.A.R.V.I.S.2.0\n```\n\n### 2ļøāƒ£ **Install Dependencies**\n\n```bash\n pip install -r requirements.txt\n```\n\n---\n\n## šŸš€ Running the Application\n\n### **Start the Program**\n\n```bash\n streamlit run ui.py\n```\n\n---\n\n## šŸ”„ **Function Calling Methods**\n\n### šŸ”¹ **Primary: Gemini AI-Based Function Execution**\n\nšŸš€ Transitioned to **Gemini AI-powered function calling**, allowing multiple **function calls simultaneously** for better efficiency! āš™ļø If Gemini AI fails to generate function calls, the system automatically falls back to an **Ollama-based model** for reliable execution.Ā \n\nšŸ”¹ **AI Model Used**: **Gemini AI** šŸ§ \\\nāœ… Higher accuracy āœ… Structured data processing āœ… Reliable AI-driven interactions\n---\n\n## šŸ“– **RAG-Based Knowledge System**\n\nšŸ’” **Retrieval-Augmented Generation (RAG)** dynamically loads relevant markdown-based knowledge files based on the queried topic, **reducing hallucinations and improving response accuracy**.\n\n---\n\n## šŸ“± **ADB Integration for Phone Automation**\n\nšŸ”¹ Integrated **Android Debug Bridge (ADB)** to enable **voice-controlled phone automation**! šŸŽ™ļø\n\nāœ… **Make phone calls** ā˜Žļø\\\nāœ… **Open apps & toggle settings** šŸ“²\\\nāœ… **Access phone data & remote operations** šŸ› ļø\n\n### **Setting Up ADB**\n\nšŸ“Œ **Windows**\n\n```powershell\nwinget install --id=Google.AndroidSDKPlatformTools -e\n```\n\nšŸ“Œ **Linux**\n\n```bash\nsudo apt install adb\n```\n\nšŸ“Œ **Mac**\n\n```bash\nbrew install android-platform-tools\n```\n\n---\n\n## šŸ”® **Future Enhancements**\n\nāœØ **Deeper mobile integration** šŸ“±\\\nāœØ **Advanced AI-driven automation** šŸ¤–\\\nāœØ **Improved NLP-based command execution** šŸ§ \\\nāœØ **Multi-modal interactions (text + voice + image)** šŸ–¼ļø\n\nšŸš€ **Stay tuned for future updates!** šŸ”„\n\n```markdown\n## Gemini Model Comparison\n\nThe following table provides a comparison of various Gemini models with respect to their rate limits:\n\n| Model | RPM | TPM | RPD |\n|------------------------------------- |-----:|----------:| -----:|\n| **Gemini 2.0 Flash** | 15 | 1,000,000 | 1,500 |\n| **Gemini 2.0 Flash-Lite Preview** | 30 | 1,000,000 | 1,500 |\n| **Gemini 2.0 Pro Experimental 02-05** | 2 | 1,000,000 | 50 |\n| **Gemini 2.0 Flash Thinking Experimental** | 10 | 4,000,000 | 1,500 |\n| **Gemini 1.5 Flash** | 15 | 1,000,000 | 1,500 |\n| **Gemini 1.5 Flash-8B** | 15 | 1,000,000 | 1,500 |\n| **Gemini 1.5 Pro** | 2 | 32,000 | 50 |\n| **Imagen 3** | -- | -- | -- |\n```\n\n### Explanation:\n- **RPM**: Requests per minute \n- **TPM**: Tokens per minute \n- **RPD**: Requests per day \n\n```\nThe focus of project is mostly on using small model and free (api) models , get accurate agentic behaviours , to run these on low spec systems to.\n```\n" }, { "path": "requirements.txt", "content": "aiofiles==24.1.0\naiohappyeyeballs==2.4.6\naiohttp==3.11.12\naiosignal==1.3.2\naltair==5.5.0\nannotated-types==0.7.0\nanyio==4.6.2.post1\nasgiref==3.8.1\nasttokens==3.0.0\nasync-timeout==4.0.3\nattrs==25.1.0\nbackoff==2.2.1\nbcrypt==4.2.1\nbeautifulsoup4==4.12.3\nblinker==1.9.0\nBrotli==1.1.0\nbuild==1.2.2.post1\ncachetools==5.5.1\ncertifi==2024.8.30\ncffi==1.17.1\nchardet==5.2.0\ncharset-normalizer==3.4.0\nchroma-hnswlib==0.7.6\nchromadb==0.6.3\nclick==8.1.8\ncoloredlogs==15.0.1\ncontourpy==1.3.0\ncryptography==44.0.1\ncycler==0.12.1\ndataclasses-json==0.6.7\ndecorator==5.1.1\ndeepsearch-glm==1.0.0\nDeprecated==1.2.18\ndill==0.3.9\ndistro==1.9.0\ndocling==2.22.0\ndocling-core==2.18.1\ndocling-ibm-models==3.3.2\ndocling-parse==3.3.1\nduckduckgo_search==7.4.2\ndurationpy==0.9\neasyocr==1.7.2\nemoji==2.14.1\net_xmlfile==2.0.0\neval_type_backport==0.2.2\nexceptiongroup==1.2.2\nexecuting==2.2.0\nfaiss-cpu==1.10.0\nfastapi==0.115.8\nfilelock==3.17.0\nfiletype==1.2.0\nflatbuffers==25.2.10\nfonttools==4.57.0\nfrozenlist==1.5.0\nfsspec==2025.2.0\nfuzzywuzzy==0.18.0\n#genai==2.1.0\ngeographiclib==2.0\ngeopy==2.4.1\ngitdb==4.0.12\nGitPython==3.1.44\ngoogle-ai-generativelanguage==0.6.15\ngoogle-api-core==2.24.1\ngoogle-api-python-client==2.161.0\ngoogle-auth==2.38.0\ngoogle-auth-httplib2==0.2.0\ngoogle-genai==1.2.0\ngoogle-generativeai==0.8.4\ngoogleapis-common-protos==1.67.0\ngrpcio==1.70.0\ngrpcio-status==1.70.0\ngTTS==2.5.4\nh11==0.14.0\nh2==4.2.0\nhpack==4.1.0\nhtml5lib==1.1\nhttpcore==1.0.7\nhttplib2==0.22.0\nhttptools==0.6.4\nhttpx==0.28.1\nhttpx-sse==0.4.0\nhuggingface-hub==0.28.1\nhumanfriendly==10.0\nhyperframe==6.1.0\nidna==3.10\nimageio==2.37.0\nimportlib_metadata==8.5.0\nimportlib_resources==6.5.2\nipython==8.18.1\njedi==0.19.2\nJinja2==3.1.5\njiter==0.8.0\njoblib==1.4.2\njsonlines==3.1.0\njsonpatch==1.33\njsonpointer==3.0.0\njsonref==1.1.0\njsonschema==4.23.0\njsonschema-specifications==2024.10.1\nkeyboard==0.13.5\nkiwisolver==1.4.7\nkubernetes==32.0.0\nlangchain==0.3.18\nlangchain-community==0.3.17\nlangchain-core==0.3.35\nlangchain-ollama==0.2.3\nlangchain-text-splitters==0.3.6\nlangdetect==1.0.9\nlangsmith==0.3.8\nlatex2mathml==3.77.0\nlazy_loader==0.4\nLevenshtein==0.27.1\nlxml==5.3.1\nMarkdown==3.7\nmarkdown-it-py==3.0.0\nmarko==2.1.2\nMarkupSafe==3.0.2\nmarshmallow==3.26.1\nmatplotlib==3.9.4\nmatplotlib-inline==0.1.7\nmdurl==0.1.2\nmmh3==5.1.0\nmonotonic==1.6\nmpire==2.10.2\nmpmath==1.3.0\nmultidict==6.1.0\nmultiprocess==0.70.17\nmypy-extensions==1.0.0\nnarwhals==1.34.1\nnest-asyncio==1.6.0\nnetworkx==3.2.1\nninja==1.11.1.3\nnltk==3.9.1\nnumpy==1.26.4\noauthlib==3.2.2\nolefile==0.47\nollama==0.4.7\nonnxruntime==1.19.2\n#openai==0.27.10\nopencv-python-headless==4.11.0.86\nopenpyxl==3.1.5\nopentelemetry-api==1.30.0\nopentelemetry-exporter-otlp-proto-common==1.30.0\nopentelemetry-exporter-otlp-proto-grpc==1.30.0\nopentelemetry-instrumentation==0.51b0\nopentelemetry-instrumentation-asgi==0.51b0\nopentelemetry-instrumentation-fastapi==0.51b0\nopentelemetry-proto==1.30.0\nopentelemetry-sdk==1.30.0\nopentelemetry-semantic-conventions==0.51b0\nopentelemetry-util-http==0.51b0\norjson==3.10.15\noverrides==7.7.0\npackaging==24.2\npandas==2.2.3\nparso==0.8.4\npdfminer.six==20240706\npexpect==4.9.0\npillow==11.1.0\nposthog==3.13.0\nprompt_toolkit==3.0.50\npropcache==0.2.1\nproto-plus==1.26.0\nprotobuf==5.29.3\npsutil==7.0.0\nptyprocess==0.7.0\npure_eval==0.2.3\npyarrow==19.0.1\npyasn1==0.6.1\npyasn1_modules==0.4.1\nPyAudio==0.2.14\npyclipper==1.3.0.post6\npycparser==2.22\npydantic==2.10.6\npydantic-settings==2.7.1\npydantic_core==2.27.2\npydeck==0.9.1\nPygments==2.19.1\npyobjc-core==11.1 ; sys_platform == \"darwin\"\npyobjc-framework-Cocoa==11.1 ; sys_platform == \"darwin\"\npyparsing==3.2.1\npypdf==5.3.0\npypdfium2==4.30.1\nPyPika==0.48.9\npyproject_hooks==1.2.0\npython-bidi==0.6.3\npython-dateutil==2.9.0.post0\npython-docx==1.1.2\npython-dotenv==1.0.1\npython-iso639==2025.2.8\npython-Levenshtein==0.27.1\npython-magic==0.4.27\npython-oxmsg==0.0.2\npython-pptx==1.0.2\npyttsx3==2.98\npytube==15.0.0\npytz==2025.1\nPyYAML==6.0.2\nRapidFuzz==3.12.1\nreferencing==0.36.2\nregex==2024.11.6\nrequests==2.32.3\nrequests-oauthlib==2.0.0\nrequests-toolbelt==1.0.0\nrich==13.9.4\nrpds-py==0.22.3\nrsa==4.9\nRtree==1.3.0\nsafetensors==0.5.2\nscikit-image==0.24.0\nscipy==1.13.1\nsemchunk==2.2.2\nshapely==2.0.7\nshellingham==1.5.4\nsix==1.17.0\nsmmap==5.0.2\nsniffio==1.3.1\nsocksio==1.0.0\nsoupsieve==2.6\nSpeechRecognition==3.12.0\nSQLAlchemy==2.0.38\nstack-data==0.6.3\nstarlette==0.45.3\nstreamlit==1.44.1\nsympy==1.13.1\ntabulate==0.9.0\ntenacity==9.0.0\ntifffile==2024.8.30\ntokenizers==0.21.0\ntoml==0.10.2\ntomli==2.2.1\ntorch==2.6.0\ntorchvision==0.21.0\ntornado==6.4.2\ntqdm==4.67.1\ntraitlets==5.14.3\ntransformers==4.48.3\ntyper==0.12.5\ntyping-inspect==0.9.0\ntyping_extensions==4.12.2\ntzdata==2025.1\nunstructured==0.16.20\nunstructured-client==0.30.0\nuritemplate==4.1.1\nurllib3==2.2.3\nuvicorn==0.34.0\nuvloop==0.21.0\nwatchfiles==1.0.4\nwcwidth==0.2.13\nwebencodings==0.5.1\nwebsocket-client==1.8.0\nwebsockets==14.2\nwrapt==1.17.2\nXlsxWriter==3.2.2\nyarl==1.18.3\nzipp==3.21.0\nzstandard==0.23.0" }, { "path": "src/BRAIN/RAG.py", "content": "import os\nimport shutil\nimport tempfile\nfrom pathlib import Path\n\nfrom docling.datamodel.base_models import InputFormat\nfrom docling.datamodel.pipeline_options import PdfPipelineOptions\nfrom docling.document_converter import DocumentConverter, PdfFormatOption, WordFormatOption, SimplePipeline\nfrom langchain_community.document_loaders import UnstructuredMarkdownLoader\nfrom langchain.text_splitter import RecursiveCharacterTextSplitter\nfrom langchain_ollama import OllamaEmbeddings, OllamaLLM\nfrom langchain_community.vectorstores import FAISS\nfrom langchain.chains import ConversationalRetrievalChain\nfrom langchain_core.runnables.history import RunnableWithMessageHistory\nfrom langchain_community.chat_message_histories.in_memory import ChatMessageHistory\nfrom src.FUNCTION.Tools.get_env import EnvManager\n\n\nclass RAGPipeline:\n def __init__(self):\n self.embedding_model = EnvManager.load_variable(\"Embedding_model\")\n self.rag_model = EnvManager.load_variable(\"Rag_model\")\n self.MAX_MESSAGES_PER_SESSION = 20\n self.qa_chain = None\n self.memory_store = {} # In-memory store for session memory\n\n def get_paths(self, subject: str):\n subject_clean = subject.lower().strip().replace(\" \", \"_\")\n doc_path = Path(f\"./DATA/RAWKNOWLEDGEBASE/{subject_clean}_data.pdf\")\n md_path = Path(f\"./DATA/KNOWLEDGEBASE/{subject_clean}_data_converted.md\")\n vectorstore_path = Path(f\"./DATA/VECTORSTORES/{subject_clean}_vectorstore.pkl\")\n return doc_path, md_path, vectorstore_path\n\n def get_document_format(self, file_path) -> InputFormat:\n ext = Path(file_path).suffix.lower()\n return {\n '.pdf': InputFormat.PDF,\n '.docx': InputFormat.DOCX,\n '.doc': InputFormat.DOCX,\n '.pptx': InputFormat.PPTX,\n '.html': InputFormat.HTML,\n '.htm': InputFormat.HTML\n }.get(ext, None)\n\n def convert_document_to_markdown(self, subject: str) -> bool:\n try:\n doc_path, md_path, _ = self.get_paths(subject)\n if not doc_path.exists():\n print(f\"No document found: {doc_path}\")\n return False\n\n doc_format = self.get_document_format(doc_path)\n if not doc_format:\n print(f\"Unsupported format: {doc_path.suffix}\")\n return False\n\n input_path = os.path.abspath(str(doc_path))\n with tempfile.TemporaryDirectory() as temp_dir:\n temp_input = os.path.join(temp_dir, os.path.basename(input_path))\n shutil.copy2(input_path, temp_input)\n\n pipeline_options = PdfPipelineOptions(do_ocr=True, do_table_structure=True)\n converter = DocumentConverter(\n allowed_formats=[doc_format],\n format_options={\n doc_format: PdfFormatOption(pipeline_options=pipeline_options),\n InputFormat.DOCX: WordFormatOption(pipeline_cls=SimplePipeline)\n }\n )\n\n conv_result = converter.convert(temp_input)\n if not conv_result or not conv_result.document:\n print(f\"Conversion failed for: {doc_path}\")\n return False\n\n markdown = conv_result.document.export_to_markdown()\n md_path.parent.mkdir(parents=True, exist_ok=True)\n with open(md_path, \"w\", encoding=\"utf-8\") as f:\n f.write(markdown)\n return True\n except Exception as e:\n print(f\"[Conversion Error] {e}\")\n return False\n\n def load_or_create_vectorstore(self, subject: str):\n try:\n _, md_path, vectorstore_path = self.get_paths(subject)\n vectorstore_path.parent.mkdir(parents=True, exist_ok=True)\n embeddings = OllamaEmbeddings(model=self.embedding_model)\n\n if vectorstore_path.exists():\n print(f\"Loading existing vectorstore from: {vectorstore_path}\")\n return FAISS.load_local(str(vectorstore_path), embeddings, allow_dangerous_deserialization=True)\n\n print(f\"Creating vectorstore for: {md_path}\")\n loader = UnstructuredMarkdownLoader(str(md_path))\n documents = loader.load()\n\n splitter = RecursiveCharacterTextSplitter(chunk_size=500, chunk_overlap=100)\n chunks = splitter.split_documents(documents)\n\n vectorstore = FAISS.from_documents(chunks, embeddings)\n vectorstore.save_local(str(vectorstore_path))\n return vectorstore\n except Exception as e:\n print(f\"[Vectorstore Error] {e}\")\n return None\n\n def get_memory(self, session_id: str):\n if session_id not in self.memory_store:\n self.memory_store[session_id] = ChatMessageHistory()\n\n history = self.memory_store[session_id]\n\n # Auto-reset memory if too long\n if len(history.messages) > self.MAX_MESSAGES_PER_SESSION:\n print(f\"[INFO] Memory for session '{session_id}' exceeded {MAX_MESSAGES_PER_SESSION} messages. Resetting.\")\n history.clear()\n\n return history\n \n def setup_chain(self, subject: str):\n try:\n _, md_path, _ = self.get_paths(subject)\n\n if not md_path.exists():\n if not self.convert_document_to_markdown(subject):\n return None\n\n vectorstore = self.load_or_create_vectorstore(subject)\n if not vectorstore:\n return None\n\n llm = OllamaLLM(model=self.rag_model, temperature=0)\n\n base_chain = ConversationalRetrievalChain.from_llm(\n llm=llm,\n retriever=vectorstore.as_retriever(search_kwargs={\"k\": 2}),\n return_source_documents=False\n )\n\n self.qa_chain = RunnableWithMessageHistory(\n base_chain,\n get_session_history=self.get_memory,\n input_messages_key=\"question\",\n history_messages_key=\"chat_history\"\n )\n return self.qa_chain\n except Exception as e:\n print(f\"[QA Chain Error] {e}\")\n return None\n\n def ask(self, qa_chain, question: str, session_id: str = \"default\") -> str:\n if not qa_chain:\n print(\"QA chain not initialized.\")\n return \"No QA chain available.\"\n try:\n result = qa_chain.invoke({\"question\": question}, config={\"configurable\": {\"session_id\": session_id}})\n return result.get(\"answer\", \"No answer found.\")\n except Exception as e:\n return f\"Error: {e}\"\n\n def interactive_chat(self, subject: str):\n if not self.setup_chain(subject):\n print(\"Could not set up RAG chain.\")\n return\n print(\"\\nChat with the RAG model. Type 'exit' to quit.\\n\")\n while True:\n question = input(\"You: \")\n if question.lower() in {\"exit\", \"quit\", \"bye\"}:\n print(\"Goodbye!\")\n break\n print(\"AI:\", self.ask(self.qa_chain, question, session_id=\"default\"))\n\n\nif __name__ == \"__main__\":\n rag = RAGPipeline()\n rag.interactive_chat(subject=\"Disaster\") # Replace with your actual subject\n" }, { "path": "src/BRAIN/chat_with_ai.py", "content": "\nimport json\nfrom pathlib import Path\nfrom langchain_ollama import ChatOllama\nfrom langchain_ollama import OllamaEmbeddings\nfrom langchain_community.vectorstores import FAISS\nfrom src.FUNCTION.Tools.get_env import EnvManager\nimport datetime\nimport math\nfrom fuzzywuzzy import fuzz\n\n\nclass PersonalChatAI:\n HISTORY_FILE_PATH = \"./DATA/chat_history.json\"\n AI_MODEL = EnvManager.load_variable(\"Chat_model\")\n EMBEDDING_MODEL = EnvManager.load_variable(\"Embedding_model\")\n SCORE_THRESHOLD = 0.6 # Adjust this threshold as needed\n MAX_HISTORY_SIZE = 100 # Limit history size\n\n def __init__(self):\n self.llm = ChatOllama(model=self.AI_MODEL, temperature=0)\n\n def get_current_timestamp(self):\n return datetime.datetime.now().isoformat()\n\n def load_chat_history(self):\n if Path(self.HISTORY_FILE_PATH).exists():\n with open(self.HISTORY_FILE_PATH, \"r\", encoding=\"utf-8\") as file:\n return json.load(file)\n return []\n\n def save_chat_history(self, history):\n with open(self.HISTORY_FILE_PATH, \"w\", encoding=\"utf-8\") as file:\n json.dump(history, file, indent=4)\n\n def ask_ai_importance(self, prompt: str) -> bool:\n \"\"\"Ask AI if the chat message is important.\"\"\"\n llm = ChatOllama(model=self.AI_MODEL, temperature=0, max_token=50)\n\n system_prompt = \"\"\"\n You are an AI that determines whether a message contains personally significant information or emotional expression.\n\n You must respond ONLY with \"yes\" or \"no\".\n\n Here are some examples:\n\n User: \"My name is Ravi and I live in Bangalore.\" \n AI: yes\n\n User: \"The weather is nice today.\" \n AI: no\n\n User: \"I'm feeling really anxious lately.\" \n AI: yes\n\n User: \"I love the book Homo Sapiens, it's my favorite.\" \n AI: yes\n\n User: \"Can you help me with a math problem?\" \n AI: no\n \"\"\"\n\n messages = [\n {\"role\": \"system\", \"content\": system_prompt},\n {\"role\": \"user\", \"content\": f\"Is this conversation important? {prompt}\"},\n ]\n\n response = llm.invoke(messages)\n return \"yes\" in response.content.strip().lower()\n\n def store_important_chat(self, prompt: str, response: str, threshold=80):\n \"\"\"Store chat in history if AI deems it important.\"\"\"\n if self.ask_ai_importance(prompt):\n cur_date_time = self.get_current_timestamp()\n history = self.load_chat_history()\n for entry in history:\n if \"user\" in entry:\n similarity_score = max(\n fuzz.token_sort_ratio(prompt, entry[\"user\"]),\n fuzz.token_set_ratio(prompt, entry[\"user\"])\n )\n\n if similarity_score >= threshold:\n entry[\"assistant\"] = response\n entry[\"timestamp\"] = cur_date_time\n break\n else:\n history.append({\"user\": prompt, \"assistant\": response, \"timestamp\": cur_date_time})\n\n if len(history) > self.MAX_HISTORY_SIZE:\n history.pop(0)\n\n self.save_chat_history(history)\n\n def distance_to_similarity_inverted(self, distance, scale=1.0):\n \"\"\"Sigmoid-based similarity mapping with inverted distance.\"\"\"\n return 1 / (1 + math.exp(-distance * scale))\n\n def semantic_search(self, query: str):\n history = self.load_chat_history()\n if not history:\n return []\n\n embedding = OllamaEmbeddings(model=self.EMBEDDING_MODEL)\n\n combined_map = {\n item[\"user\"] + \" \" + item[\"assistant\"]: item\n for item in history if \"user\" in item and \"assistant\" in item\n }\n\n vectorstore = FAISS.from_texts(\n texts=list(combined_map.keys()),\n embedding=embedding,\n )\n\n results_with_scores = vectorstore.similarity_search_with_score(query, k=7)\n filtered_results = []\n for result, score in results_with_scores:\n similarity_score = self.distance_to_similarity_inverted(score)\n if similarity_score >= self.SCORE_THRESHOLD:\n filtered_results.append(combined_map[result.page_content])\n return filtered_results\n\n def message_management(self, query):\n system_prompt = \"\"\"\n You are an AI assistant that remembers important details from previous conversations.\n When a user shares personal information (like their name, preferences, or interests), you should recall and use that naturally in responses ā€” like a thoughtful friend would. \n Avoid being overly formal or generic.\n Be warm, conversational, and use their name where appropriate.\"\"\"\n\n messages = [{\"role\": \"system\", \"content\": system_prompt}]\n relevant_chats = self.semantic_search(query)\n\n if relevant_chats:\n for chat in relevant_chats:\n messages.append({\"role\": \"user\", \"content\": chat[\"user\"]})\n messages.append({\"role\": \"assistant\", \"content\": chat[\"assistant\"]})\n\n messages.append({\"role\": \"user\", \"content\": query})\n return messages\n\n\n\n\n def personal_chat_ai(self, first_query: str, max_token: int = 2000):\n \"\"\"Chat system with persistent history and semantic retrieval.\"\"\"\n try:\n query = first_query\n messages = self.message_management(query)\n llm = ChatOllama(model=self.AI_MODEL, temperature=0, max_token=max_token)\n\n while True:\n store = len(messages) < 100\n response_stream = llm.stream(messages)\n response_content = \"\"\n\n print(\"AI:\", end=\" \")\n for chunk in response_stream:\n text = chunk.content\n print(text, end=\"\", flush=True)\n response_content += text\n print()\n\n if store:\n self.store_important_chat(query, response_content)\n\n query = input(\"YOU: \")\n if query.lower() in {\"exit\", \"end\"}:\n break\n\n messages = self.message_management(query)\n\n except Exception as e:\n print(f\"An error occurred: {e}\")\n return None\n return True\n\n\n\nif __name__ == \"__main__\":\n print(\"AI Chat Initialized. Type 'exit' to stop.\")\n first_query = input(\"YOU: \")\n chat_bot = PersonalChatAI()\n chat_bot.personal_chat_ai(first_query)\n print(\"Chat session ended.\")\n" }, { "path": "src/BRAIN/code_gen.py", "content": "\nimport logging\nimport os\nimport pandas as pd\nimport re\nfrom langchain_ollama import ChatOllama\nfrom src.FUNCTION.Tools.get_env import EnvManager\nfrom src.FUNCTION.Tools.code_exec import CodeExecutor\nfrom google import genai\n\nlogging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(module)s - %(message)s')\nlogger = logging.getLogger(__name__)\n\n\nclass CodeRefactorAssistant:\n def __init__(self):\n self.genai_key = EnvManager.load_variable(\"genai_key\")\n self.local_model = EnvManager.load_variable(\"Text_to_info_model\")\n self.execute_code_with_dependencies = CodeExecutor()\n def provide_file_details(self, path: str) -> str:\n logger.info(f\"Providing details for file: {path}\")\n try:\n df = pd.read_csv(path)\n details = [\n f\"File Path: {path}\",\n f\"File Size: {df.memory_usage(deep=True).sum() / (1024 ** 2):.2f} MB\",\n f\"Shape (rows, columns): {df.shape}\",\n f\"Column Names: {df.columns.tolist()}\",\n f\"Data Types:\\n{df.dtypes.to_string()}\",\n f\"First 5 rows:\\n{df.head().to_string(index=False)}\",\n f\"Missing values:\\n{df.isnull().sum().to_string()}\",\n f\"Summary statistics:\\n{df.describe(include='all').to_string()}\",\n f\"Unique values per column:\\n{df.nunique().to_string()}\",\n f\"Sample value types per column:\\n{df.iloc[0].to_dict()}\"\n ]\n return \"\\n\\n\".join(details)\n except FileNotFoundError:\n logger.error(f\"File not found at path: {path}\")\n return \"\"\n except Exception as e:\n logger.error(f\"Error while processing file {path}: {e}\", exc_info=True)\n return \"\"\n\n def extract_python_code(self, text):\n pattern = r\"```python\\s*(.*?)\\s*```\"\n match = re.search(pattern, text, re.DOTALL)\n if match:\n return match.group(1).strip()\n logger.info(\"No Python code found in the text.\")\n return \"\"\n\n def generate_refactor_prompt(self, code: str, error: str, file_description: str):\n return f\"\"\"\n You are an expert Python code refactor assistant. The user has provided code that generated an error during execution.\n\n File Description:\n {file_description}\n\n Code:\n {code}\n\n Error:\n {error}\n\n Instructions:\n - Analyze and refactor to resolve the issue.\n - Use pandas, numpy, matplotlib correctly.\n - Handle file-related errors.\n - Refactor to be functional and error-free.\"\"\"\n\n def gem_refactor_code(self, code: str, file_path: str, max_attempts=3):\n for attempt in range(1, max_attempts + 1):\n logger.info(f\"Gemini refactor attempt {attempt}/{max_attempts}\")\n\n if not code:\n return {\"error\": \"No code provided for execution.\"}\n\n exec_info = self.execute_code_with_dependencies.execute_code(code)\n if not exec_info.get(\"error\"):\n logger.info(\"Code executed successfully.\")\n return exec_info # āœ… Successful execution\n\n # Prepare for next attempt: describe file and generate a new prompt\n file_desc = self.provide_file_details(file_path)\n prompt = self.generate_refactor_prompt(code, exec_info[\"error\"], file_desc)\n\n try:\n client = genai.Client(api_key=self.genai_key)\n response = client.models.generate_content(model='gemini-2.0-flash', contents=prompt)\n\n if hasattr(response, \"text\"):\n code = self.extract_python_code(response.text)\n else:\n logger.warning(\"Gemini response has no 'text' attribute.\")\n return {\"error\": \"Gemini response format unexpected.\"}\n except Exception as e:\n logger.error(\"Gemini API failed\", exc_info=True)\n return {\"error\": str(e)}\n\n return {\"error\": f\"Max attempts reached. Last error: {exec_info['error']}\"}\n\n def gem_text_to_code(self, user_prompt: str, file_path: str):\n logger.info(\"Generating code from Gemini\")\n if not os.path.exists(file_path):\n raise FileNotFoundError(f\"{file_path} not found!\")\n\n file_desc = self.provide_file_details(file_path)\n prompt = f\"\"\"\n You are a Python data analysis assistant. The user has provided a CSV file at path: '{file_path}'.\n\n Use pandas, numpy, matplotlib.\n File Description:\n {file_desc}\n\n User Query:\n {user_prompt}\"\"\"\n try:\n client = genai.Client(api_key=self.genai_key)\n response = client.models.generate_content(model='gemini-2.0-flash', contents=prompt)\n code = self.extract_python_code(response.text)\n return self.gem_refactor_code(code, file_path)\n except Exception as e:\n logger.error(\"Gemini API failed\", exc_info=True)\n return {\"error\": str(e)}\n\n def local_refactor_code(self, code: str, file_path: str, max_attempts=10):\n if not code:\n return {\"error\": \"No code provided for refactoring.\"}\n\n last_error = \"Unknown error\"\n\n for attempt in range(1, max_attempts + 1):\n logger.info(f\"Local refactor attempt {attempt}/{max_attempts}\")\n\n exec_info = self.execute_code_with_dependencies.execute_code(code)\n if not exec_info.get(\"error\"):\n logger.info(\"Code executed successfully.\")\n return exec_info # āœ… Success\n\n last_error = exec_info[\"error\"]\n\n # Describe file and generate prompt\n file_desc = self.provide_file_details(file_path)\n prompt = self.generate_refactor_prompt(code, last_error, file_desc)\n\n try:\n llm = ChatOllama(model=self.local_model, temperature=0.3)\n messages = [\n {\"role\": \"system\", \"content\": prompt},\n {\"role\": \"user\", \"content\": f\"Here is the code to refactor:\\n{code}\"}\n ]\n response = llm.invoke(messages)\n code = self.extract_python_code(response)\n except Exception as e:\n logger.error(\"Local LLM failed\", exc_info=True)\n return {\"error\": str(e)}\n\n return {\"error\": f\"Max attempts reached. Last error: {last_error}\"}\n\n def local_text_to_code(self, user_prompt: str, file_path: str):\n logger.info(\"Generating code from local LLM\")\n if not os.path.exists(file_path):\n raise FileNotFoundError(f\"{file_path} not found!\")\n\n file_desc = self.provide_file_details(file_path)\n prompt = f\"\"\"\n You are a Python data analysis assistant. The user has provided a CSV file at path: '{file_path}'.\n\n Use pandas, numpy, matplotlib.\n File Description:\n {file_desc}\n\n User Query:\n {user_prompt}\"\"\"\n try:\n llm = ChatOllama(model=self.local_model, temperature=0.3)\n messages = [\n {\"role\": \"system\", \"content\": prompt},\n {\"role\": \"user\", \"content\": user_prompt}\n ]\n response = llm.invoke(messages)\n code = self.extract_python_code(response)\n return self.local_refactor_code(code, file_path)\n except Exception as e:\n logger.error(\"Local LLM failed\", exc_info=True)\n return {\"error\": str(e)}\n\n\ndef data_analysis(user_prompt: str , file_path:str):\n codeassistant = CodeRefactorAssistant()\n try:\n # Try to generate code via API\n response = codeassistant.gem_text_to_code(user_prompt , file_path)\n \n except Exception as e:\n # If API fails, log the error and fall back to local code generation\n logger.error(f\"ERROR: {e} - Falling back to local processing.\")\n response = codeassistant.local_text_to_code(user_prompt , file_path)\n\n return response\n\n" }, { "path": "src/BRAIN/gem_func_call.py", "content": "from google import genai\nfrom google.genai import types\nfrom DATA.tools import ALL_FUNCTIONS, UI_ALL_FUNCTIONS\nfrom src.FUNCTION.Tools.get_env import EnvManager\n\n\nclass GeminiFunctionCaller:\n def __init__(self):\n self.genai_key = EnvManager.load_variable(\"genai_key\")\n self.UI_ON = EnvManager.load_variable(\"UI\")\n self.client = genai.Client(api_key=self.genai_key)\n self.tools_config = self._get_tools_config()\n\n def _get_tools_config(self) -> types.GenerateContentConfig:\n tools = UI_ALL_FUNCTIONS[\"tools\"] if self.UI_ON == \"YES\" else ALL_FUNCTIONS[\"tools\"]\n return types.GenerateContentConfig(\n temperature=0,\n tools=[types.Tool(function_declarations=tools)],\n tool_config=types.ToolConfig(\n function_calling_config=types.FunctionCallingConfig(mode=\"ANY\")\n )\n )\n\n def _call_gemini(self, query: str):\n try:\n response = self.client.models.generate_content(\n model='gemini-2.0-flash',\n contents=query,\n config=self.tools_config\n )\n return response.function_calls\n except Exception as e:\n print(f\"[Gemini Error] {e}\")\n return []\n\n def generate_function_calls(self, user_query: str) -> list[dict]:\n results = []\n function_calls = self._call_gemini(user_query)\n\n for fn in function_calls:\n results.append({\n \"name\": fn.name,\n \"parameters\": fn.args\n })\n\n return results\n" }, { "path": "src/BRAIN/local_func_call.py", "content": "from typing import Union\nimport json\nimport re\nfrom langchain_ollama import ChatOllama\nfrom src.FUNCTION.Tools.get_env import EnvManager\nfrom DATA.tools import ALL_FUNCTIONS, UI_ALL_FUNCTIONS, ALL_FUNCTIONS_EXAMPLE, UI_ALL_FUNCTIONS_EXAMPLE\n\n# UI_ON = load_variable(\"UI\")\n# AVAILABLE_FUNCTION_NAMES_STRING = [func.get(\"name\") for func in ALL_FUNCTIONS.get(\"tools\")]\n# SYSTEM_MESSAGE = f\"\"\"You are an AI that determines the best function to call based on user input.\\n\\n### Available Functions:\\n{ALL_FUNCTIONS[\"tools\"] if UI_ON == \"NO\" else UI_ALL_FUNCTIONS[\"tools\"]}\\n\\n### Instructions:\\n- Choose the function name.\\n- Extract necessary arguments.\\n- **Respond ONLY in valid JSON format** as follows:\\n\\n```json\\n[\\n {{\\n \"name\": \"function_name_here\",\\n \"parameters\": {{\\n \"arg1\": \"value1\",\\n \"arg2\": \"value2\"\\n }}\\n }}\\n]\\n```\\n\\n### Examples:\\n{ALL_FUNCTIONS_EXAMPLE if UI_ON == \"NO\" else UI_ALL_FUNCTIONS_EXAMPLE}\\n```\\n\"\"\"\n\n\nUI_ON = EnvManager.load_variable(\"UI\")\nAVAILABLE_FUNCTION_NAMES_STRING = [func.get(\"name\") for func in ALL_FUNCTIONS.get(\"tools\")]\n# - If no function is relevant, return an empty list: []\n# - If the user query involves multiple actions, respond with a list of function calls in the correct order.\n\nSYSTEM_MESSAGE = f\"\"\"You are an AI that determines the best function to call based on user input.\n\n### Available Functions:\n{ALL_FUNCTIONS[\"tools\"] if UI_ON == \"NO\" else UI_ALL_FUNCTIONS[\"tools\"]}\n\n### Instructions:\n- Choose the function name.\n- Extract necessary arguments.\n- **Respond ONLY in valid JSON format** as follows:\n\n```json\n[\n {{\n \"name\": \"\",\n \"arguments\": {{\n \"arg1\": \"\",\n \"arg2\": \"\"\n }}\n }}\n]\n```\n\n### Examples:\n{ALL_FUNCTIONS_EXAMPLE if UI_ON == \"NO\" else UI_ALL_FUNCTIONS_EXAMPLE}\n```\n\"\"\"\n\nclass LocalFunctionCall:\n def __init__(self):\n self.model = EnvManager.load_variable(\"Function_call_model\")\n\n def _load_tools(self, file_path: str) -> Union[dict, list]:\n try:\n with open(file_path, \"r\") as file:\n return json.load(file)\n except FileNotFoundError:\n print(\"Error: Tools configuration file not found.\")\n return []\n except json.JSONDecodeError:\n print(\"Error: Invalid JSON format.\")\n return []\n\n def load_tools_message(self, file_path: str) -> str:\n tools = self._load_tools(file_path)\n return json.dumps(tools, indent=2)\n\n def _parse_tool_calls(self, response: str) -> Union[list, None]:\n try:\n # Remove markdown fences\n response = response.replace(\"```json\", \"\").replace(\"```\", \"\").strip()\n \n # Extract JSON-like part\n match = re.search(r'\\[.*?\\]', response, re.DOTALL)\n if not match:\n return None\n \n json_str = match.group(0)\n \n # Replace double braces with single braces\n json_str = json_str.replace(\"{{\", \"{\").replace(\"}}\", \"}\")\n \n # Parse JSON\n return json.loads(json_str)\n \n except json.JSONDecodeError as e:\n print(f\"Error parsing JSON: {e}\\nOriginal string:\\n{json_str}\")\n return None\n\n\n def create_function_call(self, user_query: str) -> Union[list, None]:\n try:\n llm = ChatOllama(model=self.model, temprature=0)\n response = llm.invoke([{\"role\": \"system\", \"content\": SYSTEM_MESSAGE}, {\"role\": \"user\", \"content\": user_query}])\n functional_response = self._parse_tool_calls(response.content)\n return [\n func for func in functional_response if func.get(\"name\", \"\").lower() in AVAILABLE_FUNCTION_NAMES_STRING\n ] if functional_response else None\n except Exception as e:\n print(f\"Error creating function call: {e}\")\n return None\n\nif __name__ == \"__main__\":\n query = \"please send email send email \"\n function_caller = LocalFunctionCall()\n response = function_caller.create_function_call(query)\n print(response)\n" }, { "path": "src/BRAIN/text_to_info.py", "content": "from langchain_ollama import ChatOllama\nfrom src.FUNCTION.Tools.get_env import EnvManager\n\nclass AIResponder:\n def __init__(self, model_name=None):\n self.model = model_name or EnvManager.load_variable(\"Text_to_info_model\")\n self.temperature = 0.3\n\n def ai_response(self, prompt: str, max_token: int = 2000) -> str:\n \"\"\"Handle creative prompts like jokes or stories.\"\"\"\n try:\n llm = ChatOllama(\n model=self.model,\n temperature=self.temperature,\n max_token=max_token\n )\n\n messages = [\n {\"role\": \"system\", \"content\": \"You are an intelligent AI system. Understand the user Query carefully and provide the most relevant Answer.\"},\n {\"role\": \"user\", \"content\": str(prompt)}\n ]\n\n response = llm.invoke(messages)\n return response.content\n\n except Exception as e:\n print(f\"An error occurred: {e}\")\n return \"Error occurred while processing your request.\"\n\ndef send_to_ai(prompt):\n AI = AIResponder()\n return AI.ai_response(prompt)\n\n# Usage Example:\n# ai_responder = AIResponder()\n# result = ai_responder.send_to_ai(\"Tell me a joke.\")\n# print(result)\n" }, { "path": "src/CONVERSATION/speech_to_text.py", "content": "import speech_recognition as sr\n\n\ndef recognize_speech():\n \"\"\"Recognize speech from the microphone.\"\"\"\n recognizer = sr.Recognizer()\n # recognizer.dynamic_energy_threshold = True\n # recognizer.energy_threshold = 30000\n # recognizer.dynamic_energy_adjustment_damping = 0.010 # less more active\n # recognizer.dynamic_energy_ratio = 1.0\n # recognizer.pause_threshold = 0.8\n # recognizer.operation_timeout = None\n # recognizer.non_speaking_duration = 0.5\n \n recognizer.energy_threshold = 3000\n recognizer.dynamic_energy_adjustment_damping = 0.07 # less more active\n recognizer.dynamic_energy_ratio = 1.5\n recognizer.pause_threshold = 0.7\n recognizer.operation_timeout = None\n recognizer.non_speaking_duration = 0.6\n \n \n # recognizer.energy_threshold = 10000 # Higher to filter out background noise\n # recognizer.dynamic_energy_adjustment_damping = 0.04 # Fast adjustments to changing noise\n # recognizer.dynamic_energy_ratio = 2.0 # Speech must be significantly louder than noise\n # recognizer.pause_threshold = 0.7 # Shorter pauses allowed\n # recognizer.operation_timeout = 10 # Stops if no speech for 10s\n # recognizer.non_speaking_duration = 0.6 # Ends sooner if background noise is stable\n\n try:\n available = sr.Microphone.list_microphone_names()\n if len(available) <= 1:\n return None \n \n with sr.Microphone() as source:\n if source:\n print(\"[=] Adjusting for ambient noise... Please wait.\")\n recognizer.adjust_for_ambient_noise(source)\n print(\"[+] Listening...\")\n audio = recognizer.listen(source)\n\n print(\"Recognizing...\")\n text = recognizer.recognize_google(audio)\n return text\n \n except sr.UnknownValueError:\n print(\"Google Speech Recognition could not understand audio\")\n\n except sr.RequestError as e:\n print(f\"Could not request results from Google Speech Recognition service; {e}\")\n\n except Exception as e:\n print(f\"An unexpected error occurred: {e}\")\n return None \n\n\n# if __name__ == \"__main__\":\n# print(\"Say 'start listening' to activate, or 'exit' to quit.\")\n# listening_mode = False\n\n# while True:\n# spoken_text = recognize_speech()\n# if spoken_text:\n# print(f\"You said: {spoken_text}\")\n\n# if \"hey jarvis\" in spoken_text.lower():\n# print(\"Listening mode activated.\")\n# listening_mode = True\n \n \n# elif \"exit\" in spoken_text.lower():\n# listening_mode = False\n# print(\"Exiting...\")\n# break\n \n# if listening_mode:\n# print(spoken_text)\n\n" }, { "path": "src/CONVERSATION/t_s.py", "content": "import threading\nimport time\nimport speech_recognition as sr\n\nrecognizer = sr.Recognizer()\n\ndef recognize_speech():\n \"\"\"Recognize speech from the microphone.\"\"\"\n recognizer.dynamic_energy_threshold = True\n recognizer.energy_threshold = 3000\n recognizer.dynamic_energy_adjustment_damping = 0.010 # More sensitive with lower values\n recognizer.dynamic_energy_ratio = 1.0\n recognizer.pause_threshold = 0.8\n recognizer.operation_timeout = None # No timeout\n recognizer.non_speaking_duration = 0.5\n\n try:\n available = sr.Microphone.list_microphone_names()\n if len(available) <= 1:\n print(\"No microphones available.\")\n return None \n \n with sr.Microphone() as source:\n print(\"[=] Adjusting for ambient noise... Please wait.\")\n recognizer.adjust_for_ambient_noise(source) # Adjust for noise\n print(\"[+] Listening...\")\n audio = recognizer.listen(source) # Start listening\n\n print(\"Recognizing...\")\n text = recognizer.recognize_google(audio) # Recognize speech\n print(f\"[Recognized] {text}\")\n return text\n \n except sr.UnknownValueError:\n print(\"Google Speech Recognition could not understand audio\")\n \n except sr.RequestError as e:\n print(f\"Could not request results from Google Speech Recognition service; {e}\")\n \n except Exception as e:\n print(f\"An unexpected error occurred: {e}\")\n \n return None \n\ndef listen_in_background():\n \"\"\"Runs speech recognition in the background.\"\"\"\n while True:\n spoken_text = recognize_speech()\n if spoken_text:\n print(f\"Detected speech: {spoken_text}\")\n time.sleep(1) # Adjust as needed\n\n# Run speech recognition in a separate thread\nlistener_thread = threading.Thread(target=listen_in_background)\nlistener_thread.daemon = True\nlistener_thread.start()\n\n# Main program keeps running while background thread listens\nwhile True:\n time.sleep(5) # Main thread sleeps, letting the listener thread do the work\n" }, { "path": "src/CONVERSATION/test_speech.py", "content": "import speech_recognition as sr\n\ndef recognize_speech():\n \"\"\"Recognize speech from the microphone.\"\"\"\n recognizer = sr.Recognizer()\n #recognizer.dynamic_energy_threshold = True\n # recognizer.energy_threshold = 3000\n # recognizer.dynamic_energy_adjustment_damping = 0.07 # less more active\n # recognizer.dynamic_energy_ratio = 1.5\n # recognizer.pause_threshold = 0.6\n # recognizer.operation_timeout = None\n # recognizer.non_speaking_duration = 0.5\n \n \n recognizer.energy_threshold = 10000 # Higher to filter out background noise\n recognizer.dynamic_energy_adjustment_damping = 0.03 # Fast adjustments to changing noise\n recognizer.dynamic_energy_ratio = 2.0 # Speech must be significantly louder than noise\n recognizer.pause_threshold = 0.8 # Shorter pauses allowed\n recognizer.operation_timeout = 10 # Stops if no speech for 10s\n recognizer.non_speaking_duration = 0.4 # Ends sooner if background noise is stable\n\n with sr.Microphone() as source:\n print(\"[=] Adjusting for ambient noise... Please wait.\")\n recognizer.adjust_for_ambient_noise(source)\n print(\"[+] Listening...\")\n audio = recognizer.listen(source)\n\n try:\n print(\"Recognizing...\")\n text = recognizer.recognize_google(audio)\n return text\n \n except sr.UnknownValueError:\n print(\"Google Speech Recognition could not understand audio\")\n return None\n except sr.RequestError as e:\n print(f\"Could not request results from Google Speech Recognition service; {e}\")\n return None\n\n# if __name__ == \"__main__\":\n# print(\"Say 'start listening' to activate, or 'exit' to quit.\")\n# listening_mode = False\n# full_text = \"\" # To store the complete speech input\n\n# while True:\n# spoken_text = recognize_speech()\n# if spoken_text:\n# print(f\"You said: {spoken_text}\")\n\n# if \"hey jarvis\" in spoken_text.lower():\n# print(\"Listening mode activated.\")\n# listening_mode = True\n# continue\n \n# elif \"stop\" in spoken_text.lower():\n# listening_mode = False\n# print(\"Listening mode deactivated...\") # Will still allow the program to continue running\n# continue \n \n \n# if listening_mode:\n# full_text += spoken_text + \" \" # Append the recognized speech\n# print(f\"Current text: {full_text}\")\n \n# if spoken_text.lower() == \"exit\":\n# print(\"Exiting...\")\n# break # Option to exit the program when 'exit' is said\n\n" }, { "path": "src/CONVERSATION/text_speech.py", "content": "\nfrom random import randint\nfrom gtts import gTTS\nfrom io import BytesIO\nimport tempfile\nimport os\nimport base64\nimport io\n\ndef text_to_speech_local(text, lang=\"en\"):\n \"\"\"Converts text to speech using gTTS and streams audio to the browser.\"\"\"\n \n # Generate TTS audio using gTTS\n tts = gTTS(text=text, lang=lang)\n \n # Create a temporary file to store audio as MP3\n with tempfile.NamedTemporaryFile(delete=False, suffix=\".mp3\") as temp_audio:\n audio_path = temp_audio.name\n\n # Save audio to the temporary file\n tts.save(audio_path)\n\n # Read the audio content as bytes into a BytesIO stream\n with open(audio_path, \"rb\") as audio_file:\n audio_bytes = audio_file.read()\n \n # Delete the temporary file after reading\n os.remove(audio_path)\n\n\n audio_base64 = base64.b64encode(audio_bytes).decode('utf-8')\n\n return audio_base64 \n\nif __name__ == \"__main__\":\n audio_io = text_to_speech_local(\"hello how are you.\")\n print(audio_io)" }, { "path": "src/CONVERSATION/text_to_speech.py", "content": "\nimport pyttsx3\nfrom random import randint\ndef speak(text:str) -> None:\n try:\n # Initialize the TTS engine\n engine = pyttsx3.init()\n # Set the speaking rate\n try:\n rate = engine.getProperty('rate')\n engine.setProperty('rate', 154) # Setting up a new speaking rate\n except Exception as e:\n print(f\"Error setting rate: {e}\")\n\n # Set the volume\n try:\n volume = engine.getProperty('volume')\n engine.setProperty('volume', 1.0) # Setting volume level between 0 and 1\n except Exception as e:\n print(f\"Error setting volume: {e}\")\n\n # Set the voice\n try:\n voices = engine.getProperty('voices')\n \n # print(\"Available Voices on macOS:\")\n # for idx, v in enumerate(voices):\n # print(f\"{idx}: Name: {v.name}, ID: {v.id}, Languages: {v.languages}\")\n \n choice = randint(0,1)\n if len(voices) >= 99: \n if choice == 1:\n engine.setProperty('voice', voices[99].id) # Set the voice by index\n else:\n engine.setProperty('voice', voices[99].id)\n else:\n engine.setProperty('voice', voices[0].id)\n except Exception as e:\n print(f\"Error setting voice: {e}\")\n\n # Speak the text\n try:\n engine.say(text)\n engine.runAndWait()\n except Exception as e:\n print(f\"Error speaking text: {e}\")\n finally:\n engine.stop()\n\n except Exception as e:\n print(f\"Error initializing TTS engine: {e}\")\n\nif __name__ == \"__main__\":\n #text = \"Welcome back, sir. Jar vis is online.\"\n text = \"Welcome back, sir. Jar-vis is online.\"\n speak(text)" }, { "path": "src/CONVERSATION/voice_text.py", "content": "import speech_recognition as sr\n\n\ndef voice_to_text(temp_audio):\n\n recognizer = sr.Recognizer()\n with sr.AudioFile(temp_audio) as source:\n audio_data = recognizer.record(source)\n \n try:\n trnanscribed_text = recognizer.recognize_google(audio_data)\n return trnanscribed_text\n except sr.UnknownValueError:\n print(\"Could not understand the audio\")\n except sr.RequestError:\n print(\"Could not request results, check your internet connection\")\n " }, { "path": "src/FUNCTION/Tools/Email_send.py", "content": "import smtplib, ssl\nfrom email.mime.multipart import MIMEMultipart\nfrom email.mime.text import MIMEText\nfrom src.FUNCTION.Tools.get_env import EnvManager\nfrom DATA.email_schema import email_prompts \nfrom src.BRAIN.text_to_info import send_to_ai\n\nclass EmailSender:\n def __init__(self):\n # Load email credentials from environment variables\n self.smtp_server = \"smtp.gmail.com\"\n self.port = 587\n self.password = EnvManager.load_variable(\"Password_email\")\n self.receiver_email = EnvManager.load_variable(\"Reciever_email\")\n self.sender_email = EnvManager.load_variable(\"Sender_email\")\n \n def initate_email(self, subject: str, email_content: str) -> bool:\n \"\"\"Send an email with the provided subject and content.\"\"\"\n html_content = f\"\"\"\n \n \n

{email_content}

\n \n \"\"\"\n \n try:\n # Prepare the email\n msg = MIMEMultipart()\n msg['From'] = self.sender_email\n msg['To'] = self.receiver_email\n msg['Subject'] = subject.strip()\n msg.attach(MIMEText(html_content, 'html'))\n \n # Set up the secure SSL context and send the email\n context = ssl.create_default_context()\n with smtplib.SMTP(self.smtp_server, self.port) as server:\n server.starttls(context=context)\n server.login(self.sender_email, self.password)\n server.sendmail(self.sender_email, self.receiver_email, msg.as_string())\n print(\"Email sent successfully...\")\n except Exception as e:\n print(f\"Error: {e}\")\n return False\n return True\n\n def email_content(self) -> dict:\n \"\"\"Generate and send an automated email based on selected template.\"\"\"\n select_template = input(\"Select an email template (job, friend, meeting, doctor, leave, product): \")\n if select_template not in email_prompts:\n print(\"[+] Invalid template selection.\")\n return {}\n \n # Fetch the selected template\n template = email_prompts[select_template]\n placeholders = {}\n \n # Collect placeholder values for the email template\n for placeholder in template['prompt'].split('{')[1:]:\n placeholder_key = placeholder.split('}')[0]\n value = input(f\"Enter value for '{placeholder_key}': \").strip()\n if not value:\n print(f\"Value for '{placeholder_key}' cannot be empty.\")\n return {}\n placeholders[placeholder_key] = value\n \n # Format the prompt with placeholders\n formatted_prompt = template['prompt'].format(**placeholders)\n \n # Display the prompt for review\n print(\"----- Start prompt -----\")\n print(formatted_prompt)\n print(\"----- End prompt -----\")\n \n # Generate email content using AI\n email_prompt = \"You are a professional email writer. Write an email based on the provided content in less than 20 words.\"\n complete_prompt = f\"{email_prompt}\\n{formatted_prompt}\"\n response = send_to_ai(complete_prompt).strip()\n \n # Generate email subject using AI\n sub_prompt = f\"Give a suitable subject for the given email: {response}. Use 3-4 words max.\"\n subject = send_to_ai(sub_prompt).strip()\n \n # Return the generated subject and content\n return {'subject': subject, 'content': response}\n\ndef send_email():\n email_sender = EmailSender()\n email_details = email_sender.email_content()\n \n if email_details:\n subject = email_details['subject']\n content = email_details['content']\n flag = email_sender.initate_email(subject, content)\n return flag \n return False\n\n# Example Usage:\nif __name__ == \"__main__\":\n email_sender = EmailSender()\n \n # Generate and send the email\n email_details = email_sender.send_email()\n \n if email_details:\n subject = email_details['subject']\n content = email_details['content']\n email_sender.initate_email(subject, content)\n" }, { "path": "src/FUNCTION/Tools/app_op.py", "content": "from src.FUNCTION.Tools.get_env import AppManager , EnvManager #load_app, check_os\nfrom os import system\n\nclass AppRunner:\n def __init__(self, name: str):\n self.name = name\n self.os_name = EnvManager.check_os()\n self.path = AppManager().load_app(name)\n\n def start_app(self) -> bool:\n \"\"\"Start the app based on the operating system.\"\"\"\n if self.os_name == \"Linux\":\n system(f'\"{self.path}\"')\n elif self.os_name == \"Darwin\":\n system(f'open \"{self.path}\"')\n elif self.os_name == \"Windows\":\n system(f'start \"{self.path}\"')\n else:\n print(\"Invalid Operating system..\")\n return False\n return True\n \n def run(self) -> str:\n \"\"\"Runs the application and returns a message indicating success or failure.\"\"\"\n if self.start_app():\n return f\"{self.name} is running now.\"\n return f\"Oops, some error occurred in opening {self.name}.\"\n\n\ndef app_runner(name:str) -> str:\n run_app = AppRunner(name)\n result = run_app.run()\n return result\n\n\n# Example usage:\nif __name__ == \"__main__\":\n app_name = input(\"Enter the name of the app to open: \")\n app_runner = AppRunner(app_name)\n result = app_runner.run()\n print(result)\n" }, { "path": "src/FUNCTION/Tools/code_exec.py", "content": "import sys\nimport subprocess\nimport importlib\nimport os\nfrom io import StringIO\n\nclass CodeExecutor:\n def __init__(self, required_libraries=None):\n \"\"\"Initialize the class with a list of required libraries.\"\"\"\n self.required_libraries = required_libraries or ['pandas', 'numpy', 'matplotlib']\n self.flag_file = './DATA/libraries_installed.txt'\n\n def get_pip_command(self):\n \"\"\"Returns the appropriate pip command based on the operating system.\"\"\"\n if sys.platform == \"win32\":\n return \"pip\" # Windows uses pip\n else:\n return \"pip3\" # macOS/Linux uses pip3\n\n def check_and_install_libraries(self):\n \"\"\"\n Checks if required libraries are installed and installs any missing ones.\n \"\"\"\n # Step 1: Check if the flag file exists\n if os.path.exists(self.flag_file) and os.path.getsize(self.flag_file) != 0:\n print(\"Libraries are already installed.\")\n return # Libraries are already installed, skip installation\n\n # If flag file does not exist, install missing libraries\n for library in self.required_libraries:\n try:\n importlib.import_module(library)\n except ImportError:\n print(f\"Library '{library}' not found. Installing...\")\n pip_command = self.get_pip_command()\n subprocess.check_call([sys.executable, \"-m\", \"pip\", \"install\", library])\n\n # Step 2: Create the flag file to indicate libraries are installed\n with open(self.flag_file, 'w') as f:\n f.write(\"Libraries installed successfully.\")\n\n def execute_code(self, code):\n \"\"\"\n Executes the dynamically generated code and ensures that required libraries are installed.\n Args:\n code (str): The Python code to execute.\n \n Returns:\n dict: Dictionary containing the result or error message.\n \"\"\"\n # Step 1: Install missing libraries if required\n self.check_and_install_libraries()\n\n # Initialize result dictionary\n result = {\n 'output': None,\n 'error': None,\n }\n\n # Step 2: Capture the output of the code execution\n old_stdout = sys.stdout\n sys.stdout = StringIO()\n\n try:\n # Step 3: Execute the code in a clean context (no interference from the global scope)\n exec_globals = {}\n exec_locals = {}\n exec(code, exec_globals, exec_locals)\n\n # Step 4: Capture the output from print statements or results\n output = sys.stdout.getvalue()\n\n # Step 5: Return the result or output\n if not output:\n result['output'] = str(exec_locals) # If no output, return the result of execution\n else:\n result['output'] = output\n \n except SyntaxError as e:\n result['error'] = f\"Syntax Error: {e.msg} on line {e.lineno}\"\n except NameError as e:\n result['error'] = f\"Name Error: {e.args}\"\n except Exception as e:\n result['error'] = f\"Error during execution: {str(e)}\"\n finally:\n sys.stdout = old_stdout # Restore the original stdout\n\n return result\n\n# Example usage:\nif __name__ == \"__main__\":\n executor = CodeExecutor() # Initialize with default libraries\n code_to_run = \"\"\"\nimport pandas as pd\nimport numpy as np\ndata = pd.DataFrame({'a': np.random.randn(100), 'b': np.random.randn(100)})\nprint(data.head())\n\"\"\"\n result = executor.execute_code(code_to_run)\n if result['error']:\n print(f\"Error: {result['error']}\")\n else:\n print(f\"Output:\\n{result['output']}\")\n" }, { "path": "src/FUNCTION/Tools/get_env.py", "content": "from dotenv import load_dotenv\nfrom os import environ\nfrom typing import Union\nimport json\nimport os\nimport platform\nfrom fuzzywuzzy import process\nfrom DATA.Domain import websites\nimport shutil\n\n\n# Load environment variables from .env file\nload_dotenv()\nAPP_JSON_PATH = \"./DATA/app.json\"\n\n\nclass EnvManager:\n \"\"\"Handles environment variable loading.\"\"\"\n \n @staticmethod\n def load_variable(variable_name: str) -> Union[str, None]:\n \"\"\"Load environment variable.\"\"\"\n try:\n variable = environ.get(variable_name.strip())\n return variable\n except Exception as e:\n print(f\"Error: {e}\")\n return None\n \n @staticmethod\n def check_os() -> str:\n \"\"\"Check the operating system and return the name.\"\"\"\n os_name = platform.system()\n if os_name == \"Windows\":\n return \"Windows\"\n elif os_name == \"Darwin\":\n return \"Darwin\"\n elif os_name == \"Linux\":\n return \"Linux\"\n else:\n return \"Unknown\"\n\n\nclass AppManager:\n \"\"\"Handles application management tasks such as checking OS, installed apps, and updating the app list.\"\"\"\n\n @staticmethod\n def is_app_installed(path: str) -> bool:\n \"\"\"Check if an application is installed by verifying its path.\"\"\"\n \n # Check if it's in the system PATH (for built-in apps like Notepad, Calculator)\n if shutil.which(path):\n return True\n \n # Check if the direct path exists (for installed applications)\n return os.path.exists(path)\n\n @staticmethod\n def get_url(website_name: str) -> str:\n \"\"\"Retrieve website URL with exact or fuzzy matching.\"\"\"\n if not website_name:\n print(\"āŒ Website name cannot be empty.\")\n return \"\"\n\n # Normalize input\n website_name = website_name.strip().lower()\n\n # Exact match\n if website_name in websites:\n return websites[website_name]\n\n # Fuzzy matching\n closest_match, score = process.extractOne(website_name, websites.keys())\n if score >= 80:\n return websites[closest_match]\n\n print(f\"āŒ Website '{website_name}' not found.\")\n return \"\"\n\n @staticmethod\n def get_app_path(app_name, app_data):\n \"\"\"Retrieve app path with exact match and fuzzy matching.\"\"\"\n # āœ… Strip and lowercase app_name (just in case)\n app_name = app_name.strip().lower()\n # āœ… Check for exact match (since app_data is already normalized)\n if app_name in app_data and AppManager.is_app_installed(app_data.get(app_name)):\n return app_data.get(app_name)\n\n # āœ… Fuzzy match for closest name in normalized keys\n closest_match, score = process.extractOne(app_name, app_data.keys())\n \n # āœ… Set a threshold for match confidence (e.g., 80)\n if score >= 80 and AppManager.is_app_installed(closest_match): # High confidence match\n return app_data[closest_match]\n \n # if no app found, fallback to website search\n link = AppManager.get_url(app_name)\n if link:\n return link\n \n # āŒ No match found\n print(f\"āŒ Application or web '{app_name}' not found.\")\n return \"\"\n\n @staticmethod\n def load_app(app_name: str) -> str:\n \"\"\"Load the path of the application from app.json.\"\"\"\n if not os.path.exists(APP_JSON_PATH) or os.path.getsize(APP_JSON_PATH) == 0:\n print(\"app.json is empty or does not exist. Fetching apps...\")\n AppManager.update_app_list()\n\n with open(APP_JSON_PATH, \"r\", encoding=\"utf-8\") as f:\n app_data = json.load(f)\n \n return AppManager.get_app_path(app_name , app_data)\n\n @staticmethod\n def update_app_list():\n \"\"\"Get the list of installed apps and store them in app.json.\"\"\"\n os_name = EnvManager.check_os()\n apps = []\n\n if os_name == \"Windows\":\n apps = AppManager.get_installed_apps_windows()\n elif os_name == \"Darwin\":\n apps = AppManager.get_installed_apps_mac()\n elif os_name == \"Linux\":\n apps = AppManager.get_installed_apps_linux()\n\n # Store in app.json\n app_dict = {app[\"name\"].lower(): app[\"path\"] for app in apps}\n \n with open(APP_JSON_PATH, \"w\", encoding=\"utf-8\") as f:\n json.dump(app_dict, f, indent=4)\n \n print(f\"{len(app_dict)} applications found and stored in app.json.\")\n\n @staticmethod\n def get_installed_apps_windows():\n \"\"\"Get installed applications on Windows.\"\"\"\n import winreg\n apps = []\n reg_paths = [\n r\"SOFTWARE\\Microsoft\\Windows\\CurrentVersion\\Uninstall\",\n r\"SOFTWARE\\WOW6432Node\\Microsoft\\Windows\\CurrentVersion\\Uninstall\",\n ]\n \n for reg_path in reg_paths:\n try:\n reg_key = winreg.OpenKey(winreg.HKEY_LOCAL_MACHINE, reg_path)\n for i in range(winreg.QueryInfoKey(reg_key)[0]):\n try:\n subkey_name = winreg.EnumKey(reg_key, i)\n subkey = winreg.OpenKey(reg_key, subkey_name)\n name, _ = winreg.QueryValueEx(subkey, \"DisplayName\")\n path, _ = winreg.QueryValueEx(subkey, \"InstallLocation\")\n if name and path:\n apps.append({\"name\": name, \"path\": path})\n except (FileNotFoundError, OSError, ValueError):\n continue\n except FileNotFoundError:\n continue\n return apps\n\n @staticmethod\n def get_installed_apps_mac():\n \"\"\"Get installed applications on macOS.\"\"\"\n app_paths = [\n \"/Applications\",\n os.path.expanduser(\"~/Applications\"),\n \"/System/Applications\", # System apps\n ]\n apps = []\n\n for path in app_paths:\n if os.path.exists(path):\n for app in os.listdir(path):\n if app.endswith(\".app\"):\n apps.append({\"name\": app.replace(\".app\", \"\"), \"path\": os.path.join(path, app)})\n\n return apps\n\n @staticmethod\n def get_installed_apps_linux():\n \"\"\"Get installed applications on Linux.\"\"\"\n import glob\n app_paths = [\"/usr/share/applications\", os.path.expanduser(\"~/.local/share/applications\")]\n apps = []\n\n for path in app_paths:\n if os.path.exists(path):\n for file in glob.glob(f\"{path}/*.desktop\"):\n with open(file, \"r\", encoding=\"utf-8\", errors=\"ignore\") as f:\n lines = f.readlines()\n name, exec_path = None, None\n for line in lines:\n if line.startswith(\"Name=\"):\n name = line.split(\"=\", 1)[1].strip()\n elif line.startswith(\"Exec=\"):\n exec_path = line.split(\"=\", 1)[1].strip()\n if name and exec_path:\n apps.append({\"name\": name, \"path\": exec_path.split()[0]})\n\n return apps\n" }, { "path": "src/FUNCTION/Tools/greet_time.py", "content": "from datetime import datetime\n\nclass TimeOfDay:\n def __init__(self):\n self.current_hour = datetime.now().hour\n\n def time_of_day(self) -> str:\n if 5 <= self.current_hour < 12:\n return \"Good morning sir!\"\n elif 12 <= self.current_hour < 17:\n return \"Good afternoon sir!\"\n elif 17 <= self.current_hour < 21:\n return \"Good evening sir!\"\n else:\n return \"Good night sir!\"\n" }, { "path": "src/FUNCTION/Tools/incog.py", "content": "import os\nfrom subprocess import run, DEVNULL\nfrom src.FUNCTION.Tools.get_env import EnvManager\n\n\nclass PrivateModeOpener:\n def __init__(self, topic: str):\n self.topic = topic\n self.search_url = f\"https://www.google.com/search?q={self.topic}\"\n self.os_name = EnvManager.check_os()\n\n def open_chrome_incognito(self) -> None:\n \"\"\"Open Chrome in Incognito mode (Windows).\"\"\"\n possible_paths = [\n r\"C:\\Program Files\\Google\\Chrome\\Application\\chrome.exe\",\n r\"C:\\Program Files (x86)\\Google\\Chrome\\Application\\chrome.exe\",\n ]\n chrome_path = next((path for path in possible_paths if os.path.exists(path)), None)\n\n if chrome_path:\n run([chrome_path, \"--incognito\", self.search_url], stdout=DEVNULL, stderr=DEVNULL)\n else:\n print(\"āŒ Chrome not found. Please install Chrome or use another browser.\")\n\n def open_firefox_private(self) -> None:\n \"\"\"Open Firefox in Private mode (Windows).\"\"\"\n possible_paths = [\n r\"C:\\Program Files\\Mozilla Firefox\\firefox.exe\",\n r\"C:\\Program Files (x86)\\Mozilla Firefox\\firefox.exe\",\n ]\n firefox_path = next((path for path in possible_paths if os.path.exists(path)), None)\n\n if firefox_path:\n run([firefox_path, \"-private-window\", self.search_url], stdout=DEVNULL, stderr=DEVNULL)\n else:\n print(\"āŒ Firefox not found. Trying Edge...\")\n self.open_edge_private()\n\n def open_edge_private(self) -> None:\n \"\"\"Open Microsoft Edge in InPrivate mode (Windows).\"\"\"\n edge_path = r\"C:\\Program Files (x86)\\Microsoft\\Edge\\Application\\msedge.exe\"\n if os.path.exists(edge_path):\n run([edge_path, \"--inprivate\", self.search_url], stdout=DEVNULL, stderr=DEVNULL)\n else:\n print(\"āŒ Edge not found. Please install a supported browser.\")\n\n def linux_firefox(self) -> None:\n \"\"\"Open Firefox in Private mode on Linux.\"\"\"\n run([\"firefox\", \"--private-window\", self.search_url], stdout=DEVNULL, stderr=DEVNULL)\n\n def incog_mode_mac(self) -> None:\n \"\"\"Open Safari in Private mode on macOS using AppleScript.\"\"\"\n applescript_code = f'''\n tell application \"Safari\"\n activate\n tell application \"System Events\"\n keystroke \"n\" using {{command down, shift down}} -- Open Private Window\n end tell\n delay 1 -- Give time to open Private Window\n tell window 1\n set current tab to (make new tab with properties {{URL:\"{self.search_url}\"}})\n end tell\n end tell\n '''\n run(['osascript', '-e', applescript_code], stdout=DEVNULL, stderr=DEVNULL)\n\n def open_in_private_mode(self) -> None:\n \"\"\"Open the specified topic in private/incognito mode.\"\"\"\n if self.os_name == \"Linux\":\n self.linux_firefox()\n\n elif self.os_name == \"Darwin\": # macOS\n self.incog_mode_mac()\n\n elif self.os_name == \"Windows\":\n try:\n self.open_chrome_incognito()\n except Exception:\n try:\n self.open_firefox_private()\n except Exception:\n self.open_edge_private()\n else:\n print(\"āŒ Unsupported Operating System.\")\n return \"Error occurred in opening in private mode\"\n \n return \"Your browser is ready in private mode.\"\n\n\n# Usage example\ndef private_mode(topic:str) -> bool:\n private_mode_opener = PrivateModeOpener(\"Artificial Intelligence\")\n result = private_mode_opener.open_in_private_mode()\n return result\n\n" }, { "path": "src/FUNCTION/Tools/internet_search.py", "content": "from src.BRAIN.text_to_info import send_to_ai\nfrom duckduckgo_search import DDGS\n\nclass DuckGoSearch:\n def __init__(self, query: str):\n self.query = query\n\n def search_query(self) -> str:\n \"\"\"Search the provided query on DuckDuckGo for quick information.\"\"\"\n results = DDGS().text(self.query, max_results=3)\n results_body = [info.get(\"body\", \"\").strip() for info in results if info.get(\"body\")]\n full_result = \"\\n\".join(results_body)\n \n return full_result\n\n def generate_answer(self, search_results: str) -> str:\n \"\"\"Generate an answer based on search results using AI.\"\"\"\n prompt = (\n \"Analyze the following search results carefully and extract the most relevant information.\"\n \"Provide a concise and accurate answer to the given query.\"\n \"\\n\\n=== Search Results ===\\n\"\n f\"{search_results}\\n\"\n \"=====================\\n\"\n f\"Query: {self.query}\\n\"\n \"Your Response:\"\n )\n answer = send_to_ai(prompt)\n return answer\n\n def execute_search(self) -> str:\n \"\"\"Execute the DuckDuckGo search and get the response.\"\"\"\n search_results = self.search_query()\n answer = self.generate_answer(search_results)\n return answer\n\n\ndef duckgo_search(query:str) -> str:\n duck_search = DuckGoSearch(query)\n answer = duck_search.execute_search()\n return answer\n" }, { "path": "src/FUNCTION/Tools/link_op.py", "content": "import webbrowser \n\ndef search_youtube(topic:str) -> None:\n \"\"\"Search YouTube for a specific topic.\"\"\"\n format_topic = \"+\".join(topic.split())\n link = f\"https://www.youtube.com/results?search_query={format_topic}\"\n try:\n webbrowser.open(link)\n except Exception as e:\n return f\"Error occured in youtube search\"\n return f\"Your {topic} search results are ready.\"\n\n\n\n" }, { "path": "src/FUNCTION/Tools/news.py", "content": "import requests\nfrom src.FUNCTION.Tools.get_env import EnvManager\nfrom typing import Union\n\nclass NewsHeadlines:\n def __init__(self, top: int = 10, country: str = \"india\"):\n self.top = top\n self.country = country\n self.api_key = EnvManager.load_variable(\"News_api\")\n\n def fetch_headlines(self) -> Union[list[str], None]:\n \"\"\"Fetch top news headlines.\"\"\"\n headlines = []\n url = (\n f'https://newsapi.org/v2/top-headlines?'\n f'q={self.country}&from=2025-04-03&to=2025-04-03&sortBy=popularity&'\n f'apiKey={self.api_key}'\n )\n \n try:\n response = requests.get(url).json()\n all_articles = response['articles']\n total_results = int(response['totalResults'])\n \n for i in range(min(self.top, total_results)):\n headline = all_articles[i]['title']\n headlines.append(headline)\n return \"\\n\".join(headlines)\n except Exception as e:\n print(f\"Error: {e}\")\n return None\n\n\n\ndef news_headlines(top=5):\n # Usage Example:\n news = NewsHeadlines(top)\n headlines = news.fetch_headlines()\n return headlines\n" }, { "path": "src/FUNCTION/Tools/phone_call.py", "content": "import subprocess\nfrom DATA.phone_details import PHONE_DIR\nfrom src.FUNCTION.Tools.get_env import EnvManager\n\nclass ADBConnect:\n def __init__(self):\n self.L_PATH_ADB = \"./src/FUNCTION/adb_connect.sh\"\n self.W_PATH_ADB = \"./src/FUNCTION/adb_connect.bat\"\n\n def adb_connect(self):\n \"\"\"Establish ADB connection over Wi-Fi.\"\"\"\n os_name = EnvManager.check_os()\n \n try:\n if os_name == \"Windows\":\n subprocess.run(self.W_PATH_ADB, shell=True, check=True)\n else:\n subprocess.run(['bash', self.L_PATH_ADB], check=True)\n except subprocess.CalledProcessError:\n return \"āŒ Failed to run ADB connect script.\"\n\n connected_devices = subprocess.run(['adb', 'devices'], capture_output=True, text=True)\n devices = connected_devices.stdout.strip().split('\\n')[1:]\n \n if not any(\"device\" in line for line in devices):\n return \"āŒ No device connected! Ensure ADB is running and the phone is connected.\"\n\n return True\n\nclass PhoneCall:\n def __init__(self, adb_connect_instance: ADBConnect):\n self.adb_connect_instance = adb_connect_instance\n\n def start_a_call(self, name: str) -> str:\n adb_response = self.adb_connect_instance.adb_connect()\n if adb_response is not True:\n return adb_response\n\n name = name.lower().strip()\n mobileNo = PHONE_DIR.get(name)\n\n if not mobileNo:\n subprocess.run(['adb', 'disconnect'], check=True)\n return f\"āŒ Contact '{name}' not found!\"\n\n try:\n subprocess.run(['adb', 'shell', 'am', 'start', '-a', 'android.intent.action.CALL', '-d', f'tel:{mobileNo}'], check=True)\n except subprocess.CalledProcessError:\n return f\"āŒ Failed to initiate call to {name}.\"\n\n # Optionally keep Wi-Fi connection alive by skipping disconnect here\n subprocess.run(['adb', 'disconnect'], check=True)\n\n return f\"šŸ“ž Phone call initiated to your friend **{name.capitalize()}**!\"\n\n# Usage Example:\ndef make_a_call(name:str) -> str:\n adb_instance = ADBConnect()\n phone_call_instance = PhoneCall(adb_instance)\n response = phone_call_instance.make_a_call(name)\n return response \n\n" }, { "path": "src/FUNCTION/Tools/random_respon.py", "content": "from random import choice\n\nclass RandomChoice:\n @staticmethod\n def random_choice(data:list) -> str:\n return choice(data)\n\n# Usage Example:\n# data_list = [\"apple\", \"banana\", \"cherry\", \"date\"]\n# random_choice_instance = RandomChoice(data_list)\n# random_item = random_choice_instance.random_choice()\n# print(random_item)\n" }, { "path": "src/FUNCTION/Tools/searxsearch.py", "content": "import requests\nfrom bs4 import BeautifulSoup\nimport os\n\nif __name__ == \"__main__\":\n from tools import Tools\nelse:\n from sources.tools.tools import Tools\n\n\n\nclass searxSearch(Tools):\n def __init__(self, base_url: str = None):\n \"\"\"\n A tool for searching a SearxNG instance and extracting URLs and titles.\n \"\"\"\n super().__init__()\n self.tag = \"web_search\"\n self.base_url = base_url or os.getenv(\"SEARXNG_BASE_URL\") # Requires a SearxNG base URL\n self.user_agent = \"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/134.0.0.0 Safari/537.36\"\n self.paywall_keywords = [\n \"Member-only\", \"access denied\", \"restricted content\", \"404\", \"this page is not working\"\n ]\n if not self.base_url:\n raise ValueError(\"SearxNG base URL must be provided either as an argument or via the SEARXNG_BASE_URL environment variable.\")\n\n def link_valid(self, link):\n \"\"\"check if a link is valid.\"\"\"\n # TODO find a better way\n if not link.startswith(\"http\"):\n return \"Status: Invalid URL\"\n \n headers = {\"User-Agent\": \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36\"}\n try:\n response = requests.get(link, headers=headers, timeout=5)\n status = response.status_code\n if status == 200:\n content = response.text.lower()\n if any(keyword in content for keyword in self.paywall_keywords):\n return \"Status: Possible Paywall\"\n return \"Status: OK\"\n elif status == 404:\n return \"Status: 404 Not Found\"\n elif status == 403:\n return \"Status: 403 Forbidden\"\n else:\n return f\"Status: {status} {response.reason}\"\n except requests.exceptions.RequestException as e:\n return f\"Error: {str(e)}\"\n\n def check_all_links(self, links):\n \"\"\"Check all links, one by one.\"\"\"\n # TODO Make it asyncromous or smth\n statuses = []\n for i, link in enumerate(links):\n status = self.link_valid(link)\n statuses.append(status)\n return statuses\n\n def execute(self, blocks: list, safety: bool = False) -> str:\n \"\"\"Executes a search query against a SearxNG instance using POST and extracts URLs and titles.\"\"\"\n if not blocks:\n return \"Error: No search query provided.\"\n\n query = blocks[0].strip()\n if not query:\n return \"Error: Empty search query provided.\"\n\n search_url = f\"{self.base_url}/search\"\n headers = {\n 'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7',\n 'Accept-Language': 'en-US,en;q=0.9',\n 'Cache-Control': 'no-cache',\n 'Connection': 'keep-alive',\n 'Content-Type': 'application/x-www-form-urlencoded',\n 'Pragma': 'no-cache',\n 'Upgrade-Insecure-Requests': '1',\n 'User-Agent': self.user_agent\n }\n data = f\"q={query}&categories=general&language=auto&time_range=&safesearch=0&theme=simple\".encode('utf-8')\n try:\n response = requests.post(search_url, headers=headers, data=data, verify=False)\n response.raise_for_status()\n html_content = response.text\n soup = BeautifulSoup(html_content, 'html.parser')\n results = []\n for article in soup.find_all('article', class_='result'):\n url_header = article.find('a', class_='url_header')\n if url_header:\n url = url_header['href']\n title = article.find('h3').text.strip() if article.find('h3') else \"No Title\"\n description = article.find('p', class_='content').text.strip() if article.find('p', class_='content') else \"No Description\"\n results.append(f\"Title:{title}\\nSnippet:{description}\\nLink:{url}\")\n if len(results) == 0:\n return \"No search results, web search failed.\"\n return \"\\n\\n\".join(results) # Return results as a single string, separated by newlines\n except requests.exceptions.RequestException as e:\n raise Exception(\"\\nSearxng search failed. did you run start_services.sh? is docker still running?\") from e\n\n def execution_failure_check(self, output: str) -> bool:\n \"\"\"\n Checks if the execution failed based on the output.\n \"\"\"\n return \"Error\" in output\n\n def interpreter_feedback(self, output: str) -> str:\n \"\"\"\n Feedback of web search to agent.\n \"\"\"\n if self.execution_failure_check(output):\n return f\"Web search failed: {output}\"\n return f\"Web search result:\\n{output}\"\n\nif __name__ == \"__main__\":\n search_tool = searxSearch(base_url=\"http://127.0.0.1:8080\")\n result = search_tool.execute([\"are dog better than cat?\"])\n print(result)\n" }, { "path": "src/FUNCTION/Tools/weather.py", "content": "from geopy.geocoders import Nominatim\nfrom requests import get\nfrom src.FUNCTION.Tools.get_env import EnvManager\nfrom src.BRAIN.text_to_info import send_to_ai\n\nclass WeatherService:\n def __init__(self, city: str):\n self.city = city\n self.api_key = EnvManager.load_variable(\"Weather_api\")\n self.geolocator = Nominatim(user_agent=\"your_app_name\")\n self.latitude, self.longitude = self.get_lat_lng(city)\n\n def get_lat_lng(self, name: str) -> tuple:\n \"\"\"Get the latitude and longitude of a given place.\"\"\"\n location = self.geolocator.geocode(name)\n if location:\n latitude = round(location.latitude, 3)\n longitude = round(location.longitude, 3)\n return latitude, longitude\n return 0, 0\n\n def weather_data(self) -> str:\n \"\"\"Get the current weather report for the city.\"\"\"\n report = {}\n url = \"https://weatherapi-com.p.rapidapi.com/current.json\"\n querystring = {\"q\": f\"{self.latitude},{self.longitude}\"}\n headers = {\n \"x-rapidapi-key\": self.api_key,\n \"x-rapidapi-host\": \"weatherapi-com.p.rapidapi.com\"\n }\n\n response = get(url, headers=headers, params=querystring)\n \n all_data = response.json()\n \n report[\"datetime\"] = all_data[\"current\"][\"last_updated\"]\n report[\"temp\"] = all_data[\"current\"][\"temp_c\"]\n report[\"condition\"] = all_data[\"current\"][\"condition\"][\"text\"]\n report[\"wind\"] = all_data[\"current\"][\"wind_kph\"]\n report[\"humidity\"] = all_data[\"current\"][\"humidity\"]\n report[\"cloud\"] = all_data[\"current\"][\"cloud\"]\n report[\"feels_like\"] = all_data[\"current\"][\"feelslike_c\"]\n report[\"uv\"] = all_data[\"current\"][\"uv\"]\n \n if report:\n summarize_text = send_to_ai(f\"{report} please summarize given data in less than 20 words without using numerical data.\")\n return summarize_text\n return \"No weather data found.\"\n\n#Usage example:\ndef weather_report(city) -> str:\n weather_service = WeatherService(city)\n weather_summary = weather_service.weather_report()\n return weather_summary\n\n" }, { "path": "src/FUNCTION/Tools/youtube_downloader.py", "content": "import re\nfrom pytube import YouTube\nfrom os import getcwd, mkdir, system\nfrom os.path import join, exists\nfrom src.FUNCTION.Tools.get_env import EnvManager\n\nclass YouTubeDownloader:\n def __init__(self, url: str):\n self.url = url\n self.save_path = EnvManager.load_variable(\"Yt_path\")\n if not self.save_path:\n raise ValueError(\"Error: No save path specified.\")\n \n def extract_id(self) -> str:\n \"\"\"Extracts the video ID from the YouTube URL.\"\"\"\n regex = r\"(?:youtube\\.com\\/(?:[^\\/\\n\\s]+\\/\\S+\\/|(?:v|e(?:mbed)?)\\/\" \\\n r\"|\\S*?[?&]v=)|youtu\\.be\\/)([a-zA-Z0-9_-]{11})\"\n match = re.search(regex, self.url)\n if not match:\n raise ValueError(\"Invalid YouTube URL\")\n return match.group(1)\n \n def download_video(self) -> str:\n \"\"\"Download a YouTube video.\"\"\"\n # Extract video ID from URL\n try:\n video_id = self.extract_id()\n except ValueError as e:\n return str(e)\n\n # Create filename\n file_name = f\"{video_id}.mp4\"\n\n # Download video\n try:\n # Create save path if it doesn't exist\n if not exists(self.save_path):\n mkdir(self.save_path)\n \n # Get the highest resolution video stream\n video = YouTube(self.url)\n highest_resolution_stream = video.streams.order_by('resolution').desc().first()\n \n # Download the video\n highest_resolution_stream.download(output_path=self.save_path, filename=file_name)\n\n return f\"Downloaded video: {file_name}\"\n except Exception as e:\n print(f\"Error downloading video: {e}\")\n return f\"Error occurred while downloading video: {e}\"\n\n# Usage example:\ndef yt_downloader(url:str) -> str:\n downloader = YouTubeDownloader(url)\n result = downloader.download_video()\n return result\n\n\n" }, { "path": "src/FUNCTION/run_function.py", "content": "from src.FUNCTION.Tools.link_op import search_youtube \nfrom src.FUNCTION.Tools.weather import weather_report\nfrom src.FUNCTION.Tools.news import news_headlines\nfrom src.FUNCTION.Tools.youtube_downloader import yt_downloader\nfrom src.FUNCTION.Tools.app_op import app_runner\nfrom src.BRAIN.text_to_info import send_to_ai \nfrom src.FUNCTION.Tools.incog import private_mode \nfrom src.FUNCTION.Tools.Email_send import send_email\nfrom src.FUNCTION.Tools.phone_call import make_a_call\nfrom src.FUNCTION.Tools.internet_search import duckgo_search\nfrom typing import Union\nfrom datetime import datetime\n\nclass FunctionExecutor:\n def __init__(self):\n self.function_map = {\n 'search_youtube': search_youtube,\n 'weather_report': weather_report,\n 'news_headlines': news_headlines,\n 'yt_download': yt_downloader,\n 'app_runner': app_runner,\n 'send_to_ai': send_to_ai,\n 'private_mode': private_mode,\n 'send_email': send_email,\n 'make_a_call': make_a_call,\n 'duckgo_search': duckgo_search,\n }\n\n def execute(self, function_call: dict) -> Union[None, dict, list]:\n \"\"\"\n Execute a function based on the function call dictionary\n\n :param function_call: Dictionary with 'name' and 'parameters' keys\n :return: Dictionary with status and result of function execution\n \"\"\"\n output = None\n func_name = function_call.get('name')\n args = function_call.get('parameters')\n\n try:\n if not func_name:\n return None\n\n func = self.function_map.get(func_name)\n\n if not func:\n print(\"[!] No matching function found.\")\n return None\n\n if args:\n all_parameters = [args[k] for k in args.keys()]\n output = func(*all_parameters)\n else:\n print(\"[*] No parameters provided.\")\n output = func()\n\n except KeyError as e:\n print(f\"[!] Missing key in function call: {e}\")\n except Exception as e:\n print(f\"[!] Error executing function: {e}\")\n\n return {\n \"status\": \"success\" if output is not None else \"failed\",\n \"function_name\": func_name,\n \"args\": args,\n \"output\": output,\n \"timestamp\": datetime.utcnow().isoformat() + \"Z\"\n }\n" }, { "path": "src/KEYBOARD/key_lst.py", "content": "from pynput.keyboard import Listener\n\n# Flag to track recording status\nis_recording = False\n\ndef on_press(key):\n global is_recording\n try:\n if key == key.up:\n if not is_recording:\n is_recording = True\n print(\"Recording started...\")\n except AttributeError:\n pass\n\ndef on_release(key):\n global is_recording\n try:\n if key == key.up:\n if is_recording:\n is_recording = False\n print(\"Recording stopped.\")\n return False # Stop listener after key release\n except AttributeError:\n pass\n\nif __name__ == \"__main__\":\n print(\"Press and hold the 'up' key to start recording. Release it to stop recording.\")\n \n # Start listening for key events\n with Listener(on_press=on_press, on_release=on_release) as listener:\n listener.join()\n" }, { "path": "src/KEYBOARD/key_prs_lst.py", "content": "import pynput.keyboard as keyboard\nimport speech_recognition as sr\nimport threading\n\n# Global flags and variables\nis_listening = False\nrecognized_text = \"\" # Variable to store recognized text\nrecognizer = sr.Recognizer()\n\ndef recognize_speech():\n \"\"\"Recognize speech from the microphone continuously.\"\"\"\n global recognized_text\n with sr.Microphone() as source:\n print(\"Listening for speech...\")\n recognizer.adjust_for_ambient_noise(source) # To adjust for ambient noise\n\n while is_listening: # Continuously listen while 'up' key is pressed\n try:\n audio = recognizer.listen(source, timeout=10) # Timeout to prevent endless listening\n print(\"Recognizing...\")\n recognized_text = recognizer.recognize_google(audio) # Store recognized text globally\n print(f\"Recognized: {recognized_text}\")\n \n except sr.UnknownValueError:\n print(\"Google Speech Recognition could not understand audio\")\n except sr.RequestError as e:\n print(f\"Could not request results from Google Speech Recognition service; {e}\")\n except sr.WaitTimeoutError:\n print(\"Listening timed out.\")\n\ndef on_press(key):\n \"\"\"Handles key press events.\"\"\"\n global is_listening\n try:\n # Only start recording if the 'up' key is pressed and we're not already listening\n if key == keyboard.Key.up and not is_listening: \n is_listening = True\n print(\"Recording started... Listening for speech.\")\n threading.Thread(target=recognize_speech, daemon=True).start() # Start speech recognition in a new thread\n except AttributeError:\n pass\n\ndef on_release(key):\n \"\"\"Handles key release events.\"\"\"\n global is_listening\n if key == keyboard.Key.up and is_listening: # Release 'up' to stop recording\n is_listening = False\n print(\"Recording stopped.\")\n print(f\"Final recorded text: {recognized_text}\") # Output the final recognized text when stopped\n\n if key == keyboard.Key.esc: # Press 'esc' to exit the program\n return False\n\nif __name__ == \"__main__\":\n print(\"Press and hold the 'up' key to start recording. Release it to stop recording.\")\n try:\n with keyboard.Listener(on_press=on_press, on_release=on_release) as listener:\n listener.join()\n except KeyboardInterrupt:\n print(\"\\nProgram interrupted. Exiting...\")\n" }, { "path": "src/VISION/gem_eye.py", "content": "import base64\nimport io\nimport os\nimport json\nimport cv2\nimport requests\nimport numpy as np\nfrom PIL import Image, ImageDraw\nfrom google import genai\nfrom src.FUNCTION.Tools.get_env import EnvManager\nimport re \n\n\nclass ImageProcessor:\n def __init__(self, image_path=\"captured_image.png\", model_name=\"gemini-2.5-flash\"):\n self.image_path = image_path\n self.require_width = 336\n self.require_height = 336\n self.model = model_name # Set model name here\n self.client = genai.Client(api_key=EnvManager.load_variable(\"genai_key\"))\n\n # ---------- Image capture and resizing ----------\n def resize_image(self, require_width=None, require_height=None) -> bool:\n require_width = require_width or self.require_width\n require_height = require_height or self.require_height\n\n try:\n with Image.open(self.image_path) as img:\n width, height = img.size\n if height <= require_height and width <= require_width:\n return True\n img = img.resize((require_width, require_height), Image.ANTIALIAS)\n img.save(self.image_path)\n print(f\"Image saved to {self.image_path}, size: {require_width}x{require_height}\")\n except Exception as e:\n print(e)\n return False\n return True\n\n def capture_image_and_save(self) -> str | None:\n cap = cv2.VideoCapture(0)\n if not cap.isOpened():\n print(\"Error: Could not open camera.\")\n return None\n try:\n ret, frame = cap.read()\n if ret:\n cv2.imwrite(self.image_path, frame)\n print(f\"Image captured and saved as {self.image_path}\")\n return self.image_path\n else:\n print(\"Error: Could not capture image.\")\n return None\n finally:\n cap.release()\n cv2.destroyAllWindows()\n\n # ---------- Basic detection ----------\n def detect_image(self , query:str) -> str | None:\n \"\"\"Detect content using the set model.\"\"\"\n if not query:\n query = \"What is this image?\"\n try:\n image = Image.open(self.image_path)\n response = self.client.models.generate_content(\n model=self.model,\n contents=[query, image]\n )\n return response.text\n except Exception as e:\n print(f\"Error: {e}\")\n return None\n\n # ---------- Object detection ----------\n def detect_objects(self, prompt=\"Detect all prominent items in the image.\") -> list | None:\n try:\n image = Image.open(self.image_path)\n config = genai.types.GenerateContentConfig(\n response_mime_type=\"application/json\"\n )\n response = self.client.models.generate_content(\n model=self.model,\n contents=[prompt, image],\n config=config\n )\n width, height = image.size\n boxes = json.loads(response.text)\n converted_boxes = []\n for box in boxes:\n y1, x1, y2, x2 = box[\"box_2d\"]\n converted_boxes.append([\n int(x1/1000*width), int(y1/1000*height),\n int(x2/1000*width), int(y2/1000*height)\n ])\n return converted_boxes\n except Exception as e:\n print(f\"Error: {e}\")\n return None\n \n def extract_segmentation_masks(self, prompt=None, output_dir=\"segmentation_results\"):\n try:\n if not self.image_path and not self.image_obj:\n print(\"No image provided.\")\n return None\n\n # Load image\n image = Image.open(self.image_path or self.image_obj)\n image.thumbnail([1024, 1024], Image.Resampling.LANCZOS)\n\n # Default prompt\n prompt = prompt or \"Give segmentation masks for prominent items.\"\n # Enforce structured JSON output\n prompt = f\"\"\"{prompt}\n Output ONLY a JSON list of segmentation masks.\n Each entry should have:\n - \"box_2d\": [y0, x0, y1, x1]\n - \"mask\": base64 PNG string\n - \"label\": descriptive text\n Do not include any extra text or explanations.\"\"\"\n\n # Prepare model config\n config = genai.types.GenerateContentConfig(\n thinking_config=genai.types.ThinkingConfig()\n )\n\n # Call the model\n response = self.client.models.generate_content(\n model=self.model,\n contents=[prompt, image],\n config=config\n )\n\n # Parse JSON safely\n try:\n items = json.loads(self._parse_json(response.text))\n except json.JSONDecodeError as e:\n print(f\"JSON parsing failed: {e}\")\n items = []\n\n os.makedirs(output_dir, exist_ok=True)\n composite_images = [] # Store overlay images for Streamlit\n\n for i, item in enumerate(items):\n if not all(k in item for k in (\"box_2d\", \"mask\", \"label\")):\n print(f\"Skipping item {i}: missing required keys\")\n continue\n\n y0, x0, y1, x1 = item.get(\"box_2d\", [0, 0, image.height, image.width])\n y0 = int(y0 / 1000 * image.size[1])\n x0 = int(x0 / 1000 * image.size[0])\n y1 = int(y1 / 1000 * image.size[1])\n x1 = int(x1 / 1000 * image.size[0])\n if y0 >= y1 or x0 >= x1:\n continue\n\n # Decode mask\n png_str = item[\"mask\"]\n if not png_str.startswith(\"data:image/png;base64,\"):\n continue\n png_str = png_str.removeprefix(\"data:image/png;base64,\")\n mask_data = base64.b64decode(png_str)\n mask = Image.open(io.BytesIO(mask_data)).resize((x1-x0, y1-y0), Image.Resampling.BILINEAR)\n\n # Create overlay\n overlay = Image.new('RGBA', image.size, (0, 0, 0, 0))\n overlay_draw = ImageDraw.Draw(overlay)\n mask_array = np.array(mask)\n color = (255, 255, 255, 200)\n for y in range(y0, y1):\n for x in range(x0, x1):\n if mask_array[y - y0, x - x0] > 128:\n overlay_draw.point((x, y), fill=color)\n\n # Save mask and overlay\n mask_filename = f\"{item['label']}_{i}_mask.png\"\n overlay_filename = f\"{item['label']}_{i}_overlay.png\"\n mask.save(os.path.join(output_dir, mask_filename))\n composite = Image.alpha_composite(image.convert('RGBA'), overlay)\n composite.save(os.path.join(output_dir, overlay_filename))\n\n composite_images.append(composite) # For Streamlit\n print(f\"Saved mask and overlay for {item['label']} to {output_dir}\")\n\n # Convert first composite image to base64 for Streamlit display\n if composite_images:\n buffered = io.BytesIO()\n composite_images[0].save(buffered, format=\"PNG\")\n img_base64 = base64.b64encode(buffered.getvalue()).decode()\n return img_base64\n\n return None\n\n except Exception as e:\n print(f\"Error in extract_segmentation_masks: {e}\")\n return None\n \n # ---------- Helper ----------\n def _parse_json(self, json_output: str):\n \"\"\"\n Extracts the first JSON array or object from a model output,\n even if there is extra text around it.\n \"\"\"\n\n\n # Remove markdown code fences if present\n json_output = re.sub(r\"```(json)?\", \"\", json_output, flags=re.IGNORECASE).strip()\n\n # Find the first JSON object or array\n start = json_output.find(\"[\")\n end = json_output.rfind(\"]\") + 1\n if start != -1 and end != -1:\n try:\n return json.loads(json_output[start:end])\n except json.JSONDecodeError as e:\n print(f\"JSON parsing failed: {e}\")\n return []\n else:\n # Try object instead of list\n start = json_output.find(\"{\")\n end = json_output.rfind(\"}\") + 1\n if start != -1 and end != -1:\n try:\n return [json.loads(json_output[start:end])]\n except json.JSONDecodeError as e:\n print(f\"JSON parsing failed: {e}\")\n return []\n return []\n\n\n # ---------- File API ----------\n def upload_image_file(self) -> genai.types.File:\n return self.client.files.upload(file=self.image_path)\n\n # ---------- Multi-image prompt ----------\n def multi_image_prompt(self, images: list, prompt: str):\n contents = [prompt]\n for img in images:\n if isinstance(img, str):\n contents.append(self.client.files.upload(file=img))\n elif isinstance(img, Image.Image):\n contents.append(img)\n response = self.client.models.generate_content(\n model=self.model,\n contents=contents\n )\n return response.text\n\n # ---------- Inline image from URL ----------\n def get_image_from_url(self, url: str) -> Image.Image:\n img_bytes = requests.get(url).content\n return Image.open(io.BytesIO(img_bytes))\n\n\ndef main():\n # Initialize processor\n processor = ImageProcessor(image_path=\"test_image.png\")\n\n # ---------- 1. Capture image ----------\n captured_path = processor.capture_image_and_save()\n if captured_path:\n print(f\"Captured image saved at: {captured_path}\")\n\n # ---------- 2. Resize image ----------\n if processor.resize_image(336, 336):\n print(f\"Image resized successfully.\")\n\n # ---------- 3. Basic detection ----------\n detection = processor.detect_image()\n if detection:\n print(\"Basic detection output:\")\n print(detection)\n\n # ---------- 4. Object detection ----------\n boxes = processor.detect_objects(prompt=\"Detect prominent objects and their bounding boxes.\")\n if boxes:\n print(\"Detected bounding boxes:\")\n for b in boxes:\n print(b)\n\n # ---------- 5. Segmentation ----------\n processor.extract_segmentation_masks(\n prompt=\"Segment all wooden and glass items in the image.\",\n output_dir=\"segmentation_results\"\n )\n\n # ---------- 6. File API upload ----------\n uploaded_file = processor.upload_image_file()\n print(f\"Uploaded file ID: {uploaded_file.id}\")\n\n # ---------- 7. Multi-image prompt ----------\n # For demonstration, using the same image twice\n multi_response = processor.multi_image_prompt(\n images=[processor.image_path, processor.image_path],\n prompt=\"Compare these two images and describe the differences.\"\n )\n print(\"Multi-image prompt response:\")\n print(multi_response)\n\n # ---------- 8. Fetch image from URL ----------\n url = \"https://via.placeholder.com/150\" # Example image URL\n image_from_url = processor.get_image_from_url(url)\n image_from_url.show()\n print(\"Fetched and displayed image from URL.\")\n\nif __name__ == \"__main__\":\n main()\n\n\n" }, { "path": "src/VISION/local_eye.py", "content": "import base64\nimport io\nimport os\nimport json\nimport cv2\nimport requests\nimport numpy as np\nfrom PIL import Image, ImageDraw\nfrom src.FUNCTION.Tools.get_env import EnvManager\nimport re\n\n# The ollama library is required for this local processor.\n# You can install it with: pip install ollama\ntry:\n import ollama\nexcept ImportError:\n raise ImportError(\"The 'ollama' library is required for LocalImageProcessor. Please install it using 'pip install ollama'.\")\n\nclass LocalImageProcessor:\n \"\"\"\n A class to process images using a local multimodal model via Ollama.\n Mirrors the functionality of ImageProcessor for a local environment.\n \"\"\"\n def __init__(self, image_path=\"captured_image.png\", model_name=\"llava:7b\"):\n \"\"\"\n Initializes the processor.\n\n Args:\n image_path (str): The default path to save/read images.\n model_name (str): The name of the local model to use with Ollama (e.g., 'llava').\n \"\"\"\n \n self.image_path = image_path\n self.require_width = 336\n self.require_height = 336\n if model_name:\n self.model = model_name\n else:\n self.model = EnvManager.load_variable(\"Image_to_text\")\n \n # Check if the Ollama server is running and the model is available\n try:\n ollama.show(model_name)\n except Exception as e:\n print(f\"Error connecting to Ollama or finding model '{model_name}'.\")\n print(\"Please ensure the Ollama server is running and you have pulled the model (e.g., 'ollama run llava').\")\n raise e\n\n # ---------- Image capture and resizing (No changes) ----------\n def resize_image(self, require_width=None, require_height=None) -> bool:\n \"\"\"Resizes the image to the specified dimensions.\"\"\"\n require_width = require_width or self.require_width\n require_height = require_height or self.require_height\n\n try:\n with Image.open(self.image_path) as img:\n # The ANTIALIAS attribute is deprecated and will be removed in Pillow 10 (2023-07-01). \n # Use Resampling.LANCZOS instead.\n resample_filter = Image.Resampling.LANCZOS if hasattr(Image, 'Resampling') else Image.ANTIALIAS\n img = img.resize((require_width, require_height), resample_filter)\n img.save(self.image_path)\n print(f\"Image saved to {self.image_path}, size: {require_width}x{require_height}\")\n except Exception as e:\n print(f\"Error during resize: {e}\")\n return False\n return True\n\n def capture_image_and_save(self) -> str | None:\n \"\"\"Captures an image from the webcam and saves it.\"\"\"\n cap = cv2.VideoCapture(0)\n if not cap.isOpened():\n print(\"Error: Could not open camera.\")\n return None\n try:\n ret, frame = cap.read()\n if ret:\n cv2.imwrite(self.image_path, frame)\n print(f\"Image captured and saved as {self.image_path}\")\n return self.image_path\n else:\n print(\"Error: Could not capture image.\")\n return None\n finally:\n cap.release()\n cv2.destroyAllWindows()\n\n # ---------- Basic detection ----------\n def detect_image(self, query: str) -> str | None:\n \"\"\"Detect content using the set local model.\"\"\"\n if not query:\n query = \"What is this image?\"\n try:\n with open(self.image_path, 'rb') as f:\n image_bytes = f.read()\n \n response = ollama.chat(\n model=self.model,\n messages=[\n {\n 'role': 'user',\n 'content': query,\n 'images': [image_bytes]\n }\n ]\n )\n return response['message']['content']\n except Exception as e:\n print(f\"Error during local detection: {e}\")\n return None\n\n # ---------- Object detection ----------\n def detect_objects(self, prompt=\"Detect all prominent items in the image.\") -> list | None:\n \"\"\"Detects objects using a text prompt to force JSON output.\"\"\"\n # Strong prompt to guide the local model to produce structured JSON\n json_prompt = f\"\"\"{prompt}. \n Analyze the provided image and identify the prominent objects.\n Your output must be ONLY a valid JSON list of objects. Do not include any text, explanation, or markdown.\n Each object in the list should be a dictionary with one key: \"box_2d\".\n The \"box_2d\" value must be a list of four integers: [y1, x1, y2, x2].\n These coordinates should be normalized to a 1000x1000 grid, where y1 < y2 and x1 < x2.\n Example format: [{{\"box_2d\": [250, 150, 750, 850]}}, {{\"box_2d\": [100, 300, 400, 700]}}]\n \"\"\"\n try:\n image = Image.open(self.image_path)\n width, height = image.size\n\n with open(self.image_path, 'rb') as f:\n image_bytes = f.read()\n\n response = ollama.chat(\n model=self.model,\n messages=[\n {\n 'role': 'user',\n 'content': json_prompt,\n 'images': [image_bytes]\n }\n ]\n )\n \n # Use the helper to parse the model's text response\n boxes_data = self._parse_json(response['message']['content'])\n \n if not boxes_data:\n print(\"Could not parse JSON from model response.\")\n return None\n \n converted_boxes = []\n for box in boxes_data:\n if \"box_2d\" in box and len(box[\"box_2d\"]) == 4:\n y1, x1, y2, x2 = box[\"box_2d\"]\n # Convert from 1000x1000 grid to actual image dimensions\n converted_boxes.append([\n int(x1 / 1000 * width), int(y1 / 1000 * height),\n int(x2 / 1000 * width), int(y2 / 1000 * height)\n ])\n return converted_boxes\n except Exception as e:\n print(f\"Error during local object detection: {e}\")\n return None\n\n def extract_segmentation_masks(self, prompt=None, output_dir=\"segmentation_results\"):\n \"\"\"This functionality is not supported by standard local VLMs like LLaVA.\"\"\"\n print(\"Warning: Segmentation mask generation is not supported by this local processor.\")\n raise NotImplementedError(\"Standard local VLMs do not generate pixel-level segmentation masks.\")\n\n # ---------- Helper (No changes) ----------\n def _parse_json(self, text_output: str) -> list:\n \"\"\"\n Extracts the first JSON list from a model output string,\n even if there is extra text or markdown fences around it.\n \"\"\"\n text_output = re.sub(r\"```(json)?\", \"\", text_output, flags=re.IGNORECASE).strip()\n\n start = text_output.find(\"[\")\n end = text_output.rfind(\"]\") + 1\n \n if start != -1 and end != -1:\n json_str = text_output[start:end]\n try:\n return json.loads(json_str)\n except json.JSONDecodeError as e:\n print(f\"JSON parsing failed on extracted text: {e}\")\n return []\n return []\n\n # ---------- File API (Not applicable to local models) ----------\n def upload_image_file(self):\n \"\"\"This functionality is not applicable for local, stateless model servers like Ollama.\"\"\"\n print(\"Warning: The concept of a File API does not apply to this local processor.\")\n raise NotImplementedError(\"Local models process images directly per request; there is no file upload API.\")\n\n # ---------- Multi-image prompt ----------\n def multi_image_prompt(self, images: list[str], prompt: str) -> str | None:\n \"\"\"Sends a prompt with multiple images to the local model.\"\"\"\n try:\n image_bytes_list = []\n for img_path in images:\n if not os.path.exists(img_path):\n print(f\"Image path does not exist: {img_path}\")\n continue\n with open(img_path, 'rb') as f:\n image_bytes_list.append(f.read())\n \n if not image_bytes_list:\n print(\"No valid images to process.\")\n return None\n\n response = ollama.chat(\n model=self.model,\n messages=[\n {\n 'role': 'user',\n 'content': prompt,\n 'images': image_bytes_list\n }\n ]\n )\n return response['message']['content']\n except Exception as e:\n print(f\"Error during multi-image prompt: {e}\")\n return None\n\n # ---------- Inline image from URL (No changes) ----------\n def get_image_from_url(self, url: str) -> Image.Image | None:\n \"\"\"Downloads an image from a URL and returns a PIL Image object.\"\"\"\n try:\n response = requests.get(url)\n response.raise_for_status() # Raise an exception for bad status codes\n img_bytes = response.content\n return Image.open(io.BytesIO(img_bytes))\n except requests.RequestException as e:\n print(f\"Error fetching image from URL {url}: {e}\")\n return None\n\n\ndef main_local():\n \"\"\"Main function to demonstrate the LocalImageProcessor.\"\"\"\n # Ensure a test image exists. Let's create a placeholder if not.\n test_image_path = \"test_image_local.png\"\n if not os.path.exists(test_image_path):\n try:\n img = Image.new('RGB', (640, 480), color = 'red')\n d = ImageDraw.Draw(img)\n d.rectangle([100, 100, 300, 300], fill='blue')\n d.text((50,50), \"Test Image\", fill=(255,255,0))\n img.save(test_image_path)\n print(f\"Created a dummy test image: {test_image_path}\")\n except Exception as e:\n print(f\"Could not create a test image: {e}\")\n return\n\n # Initialize processor with a local model name\n # Ensure you have run 'ollama pull llava' first\n try:\n processor = LocalImageProcessor(image_path=test_image_path, model_name=\"llava\")\n except Exception:\n return # Stop if initialization fails\n\n # ---------- 1. Basic detection ----------\n print(\"\\n--- 1. Basic Detection ---\")\n detection = processor.detect_image(\"Describe this image in detail.\")\n if detection:\n print(\"Basic detection output:\")\n print(detection)\n\n # ---------- 2. Object detection ----------\n print(\"\\n--- 2. Object Detection ---\")\n boxes = processor.detect_objects(prompt=\"Detect the blue square.\")\n if boxes:\n print(\"Detected bounding boxes:\")\n for b in boxes:\n print(b)\n\n # ---------- 3. Multi-image prompt ----------\n print(\"\\n--- 3. Multi-image Prompt ---\")\n # For demonstration, creating a second test image\n test_image_2_path = \"test_image_local_2.png\"\n img2 = Image.new('RGB', (640, 480), color = 'green')\n img2.save(test_image_2_path)\n \n multi_response = processor.multi_image_prompt(\n images=[processor.image_path, test_image_2_path],\n prompt=\"Describe the primary color of each image. First image, then second image.\"\n )\n if multi_response:\n print(\"Multi-image prompt response:\")\n print(multi_response)\n \n # ---------- 4. Fetch image from URL ----------\n print(\"\\n--- 4. Fetch image from URL ---\")\n url = \"[https://via.placeholder.com/150/0000FF/FFFFFF?Text=URL+Image](https://via.placeholder.com/150/0000FF/FFFFFF?Text=URL+Image)\"\n image_from_url = processor.get_image_from_url(url)\n if image_from_url:\n image_from_url.save(\"url_image.png\")\n print(\"Fetched image from URL and saved as url_image.png\")\n \n # ---------- 5. Unsupported features ----------\n print(\"\\n--- 5. Testing Unsupported Features ---\")\n try:\n processor.extract_segmentation_masks()\n except NotImplementedError as e:\n print(f\"Correctly caught expected error: {e}\")\n \n try:\n processor.upload_image_file()\n except NotImplementedError as e:\n print(f\"Correctly caught expected error: {e}\")\n\n\nif __name__ == \"__main__\":\n # You can run the original main() or the new main_local()\n # main() # For the Google GenAI Processor\n main_local() # For the Local Ollama Processor" }, { "path": "ui.py", "content": "import os\nimport json\nimport tempfile\nimport streamlit as st\nfrom langchain_ollama import ChatOllama\nimport torch\nimport io \nimport base64\n\n# ----- Custom Modules -----\nfrom src.FUNCTION.run_function import FunctionExecutor\nfrom src.BRAIN.text_to_info import send_to_ai\nfrom src.BRAIN.local_func_call import LocalFunctionCall\nfrom src.CONVERSATION.text_to_speech import speak\nfrom src.FUNCTION.Tools.random_respon import RandomChoice\nfrom src.FUNCTION.Tools.greet_time import TimeOfDay\nfrom DATA.msg import WELCOME_RESPONSES\nfrom src.BRAIN.gem_func_call import GeminiFunctionCaller\nfrom src.BRAIN.RAG import RAGPipeline\nfrom src.BRAIN.chat_with_ai import PersonalChatAI\nfrom src.CONVERSATION.text_speech import text_to_speech_local\nfrom src.CONVERSATION.voice_text import voice_to_text\nfrom src.BRAIN.code_gen import CodeRefactorAssistant\nfrom src.VISION.gem_eye import ImageProcessor \nfrom src.VISION.local_eye import LocalImageProcessor\n\n\n# # ----- Torch fix -----\nif hasattr(torch.classes, '__path__'):\n torch.classes.__path__ = []\n\n# ----- Initialize Components -----\nlocal_caller = LocalFunctionCall()\nfunc_executor = FunctionExecutor()\ntime_greeter = TimeOfDay()\ncode_assistant = CodeRefactorAssistant()\nchat_ai = PersonalChatAI()\nrag = RAGPipeline()\n\n# ----- Streamlit config -----\nos.environ[\"STREAMLIT_WATCHER_TYPE\"] = \"none\"\nUPLOAD_DIR = \".\"\nos.makedirs(UPLOAD_DIR, exist_ok=True)\n\nAI_MODEL = \"granite3.1-dense:2b\"\n\n# ----- Session Initialization -----\ndef initialize_session():\n defaults = {\n \"chat_mode\": \"normal\",\n \"chat_histories\": {\n \"normal\": [],\n \"chat_with_ai\": [],\n \"chat_with_rag\": [],\n \"data_analysis\": [],\n \"image_processing\": []\n },\n \"rag_subject\": \"\",\n \"voice_output\": False,\n \"uploaded_file_path\": None,\n \"audio_input_key_counter\": 0,\n \"image_path\": None,\n \"image_obj\": None,\n \"image_action\": None\n }\n for k, v in defaults.items():\n if k not in st.session_state:\n st.session_state[k] = v\n\ndef set_greeting():\n if \"greeted_once\" not in st.session_state:\n st.session_state.greeted_once = False\n if not st.session_state.greeted_once:\n greeting_message = f\"{time_greeter.time_of_day()}. {RandomChoice.random_choice(WELCOME_RESPONSES)}\"\n st.session_state.greeting_message = greeting_message\n st.session_state.greeted_once = True\n\n\ninitialize_session()\nset_greeting()\nif \"greeting_message\" in st.session_state:\n speak(st.session_state.greeting_message)\n del st.session_state[\"greeting_message\"]\n\n# ----- Utility Functions -----\n@st.cache_resource(show_spinner=False)\ndef load_rag_chain(subject):\n try:\n return rag.setup_chain(subject.lower().strip().replace(\" \", \"_\"))\n except:\n return None\n\n@st.cache_data(show_spinner=False)\ndef personal_chat_ai(query, max_token=2000):\n try:\n messages = chat_ai.message_management(query)\n llm = ChatOllama(model=AI_MODEL, temperature=0.3, max_token=max_token)\n response_content = \"\".join(chunk.content for chunk in llm.stream(messages))\n chat_ai.store_important_chat(query, response_content)\n return response_content\n except Exception as e:\n return f\"An error occurred: {e}\"\n\ndef data_analysis(user_prompt: str, file_path: str):\n try:\n return code_assistant.gem_text_to_code(user_prompt, file_path)\n except:\n return code_assistant.local_text_to_code(user_prompt, file_path)\n\ndef chat_with_rag_session(subject, query):\n key = subject.lower().strip().replace(\" \", \"_\")\n if f\"qa_chain_{key}\" not in st.session_state:\n st.session_state[f\"qa_chain_{key}\"] = load_rag_chain(key)\n qa_chain = st.session_state.get(f\"qa_chain_{key}\")\n return rag.ask(qa_chain, query) if qa_chain else f\"Error: Unable to load RAG chain for '{subject}'.\"\n\ndef process_command(command):\n try:\n gem_caller = GeminiFunctionCaller()\n response_list_dic = gem_caller.generate_function_calls(command)\n if not response_list_dic:\n raise ValueError(\"Empty Gemini output.\")\n except:\n response_list_dic = local_caller.create_function_call(command)\n if not response_list_dic:\n raise ValueError(\"Empty Local model output.\")\n \n results = []\n for dic in response_list_dic:\n func_name = dic.get(\"name\")\n args = dic.get(\"arguments\", {})\n speak(f\"Executing function: {func_name}\")\n try:\n res = func_executor.execute(dic)\n results.append(res)\n except Exception as e:\n results.append({\"status\": \"failed\", \"function_name\": func_name, \"args\": args, \"output\": str(e)})\n send_to_ai(f\"Respond to user's command '{command}' concisely.\")\n return results\n\ndef add_message(role, content):\n history = st.session_state.chat_histories[st.session_state.chat_mode]\n if not any(msg[\"content\"] == content and msg[\"role\"] == role for msg in history):\n history.append({\"role\": role, \"content\": content})\n\n# ----- Sidebar: Mode Selection -----\nst.sidebar.markdown(\"### šŸ” Select Chat Mode\")\nmode_display_map = {\n \"šŸ’¬ Normal\": \"normal\",\n \"šŸ§ Personal Chat\": \"chat_with_ai\",\n \"šŸ“š RAG Chat\": \"chat_with_rag\",\n \"šŸ“Š Data Analysis\": \"data_analysis\",\n \"šŸ–¼ļø Image Processing\": \"image_processing\"\n}\nselected_display = st.sidebar.selectbox(\"šŸ§  Select Chat Mode\", list(mode_display_map.keys()))\nselected_mode = mode_display_map[selected_display]\nst.session_state.chat_mode = selected_mode\n\n# Voice toggle\nst.session_state.voice_output = st.sidebar.toggle(\"šŸŽ™ļø Voice Reply\", value=st.session_state.voice_output)\n\n# RAG topic select\nif st.session_state.chat_mode == \"chat_with_rag\":\n st.session_state.rag_subject = st.sidebar.selectbox(\"šŸ“˜ Select RAG Topic\", [\n \"Disaster\", \"Finance\", \"Healthcare\", \"Artificial Intelligence\", \"Climate Change\",\n \"Cybersecurity\", \"Education\", \"Space Technology\", \"Politics\", \"History\", \"Biology\"\n ])\n\n# Data Analysis upload\nif st.session_state.chat_mode == \"data_analysis\":\n st.sidebar.markdown(\"### šŸ“¤ Upload CSV File\")\n uploaded_file = st.sidebar.file_uploader(\"Choose CSV\", type=[\"csv\"])\n if uploaded_file:\n file_path = os.path.join(UPLOAD_DIR, uploaded_file.name)\n with open(file_path, \"wb\") as f:\n f.write(uploaded_file.getbuffer())\n st.session_state.uploaded_file_path = file_path\n st.sidebar.success(f\"āœ… File saved to: `{file_path}`\")\n\n# Image Processing setup\nif st.session_state.chat_mode == \"image_processing\":\n st.sidebar.markdown(\"### šŸ–¼ļø Image Input\")\n image_source = st.sidebar.radio(\"Image Source\", [\"Upload\", \"URL\", \"Camera\"])\n try:\n processor = ImageProcessor()\n except Exception as e:\n processor = LocalImageProcessor()\n \n if image_source == \"Upload\":\n uploaded_img = st.sidebar.file_uploader(\"Upload Image\", type=[\"png\",\"jpg\",\"jpeg\"])\n if uploaded_img:\n path = os.path.join(UPLOAD_DIR, uploaded_img.name)\n with open(path, \"wb\") as f: f.write(uploaded_img.getbuffer())\n processor.image_path = path\n st.session_state.image_path = path\n st.image(path, caption=\"Uploaded Image\")\n elif image_source == \"URL\":\n url = st.sidebar.text_input(\"Image URL\")\n if url:\n img = processor.get_image_from_url(url)\n st.session_state.image_obj = img\n processor.image_obj = img\n st.image(img, caption=\"Fetched Image\")\n elif image_source == \"Camera\":\n if st.sidebar.button(\"šŸ“ø Capture Image\"):\n captured = processor.capture_image_and_save()\n if captured:\n st.session_state.image_path = captured\n processor.image_path = captured\n st.image(captured, caption=\"Captured Image\")\n\n st.session_state.image_action = st.sidebar.selectbox(\n \"Select Action\", [\"Basic Detection\", \"Object Detection\", \"Segmentation\"]\n )\n\n# ----- Current Mode Display -----\nst.markdown(f\"
šŸ§  Current Mode: {st.session_state.chat_mode.upper()}
\", unsafe_allow_html=True)\n\n# ----- Show Chat History -----\nfor msg in st.session_state.chat_histories[st.session_state.chat_mode]:\n with st.chat_message(msg[\"role\"]):\n st.markdown(msg[\"content\"], unsafe_allow_html=True)\n if \"image\" in msg and msg[\"image\"]:\n st.image(\n io.BytesIO(base64.b64decode(msg[\"image\"])),\n use_column_width=True\n )\n\n# ----- Chat Input -----\nif user_input := st.chat_input(\"Ask me anything... \"):\n st.chat_message(\"user\").markdown(user_input)\n add_message(\"user\", user_input)\n\n # --- Mode handling ---\n if st.session_state.chat_mode == \"chat_with_ai\":\n response = personal_chat_ai(user_input)\n elif st.session_state.chat_mode == \"chat_with_rag\":\n subject = st.session_state.rag_subject\n response = chat_with_rag_session(subject, user_input) if subject else \"šŸšØ Enter a subject.\"\n elif st.session_state.chat_mode == \"data_analysis\":\n if not st.session_state.uploaded_file_path:\n response = \"šŸšØ Please upload a CSV first.\"\n else:\n response = data_analysis(user_input, st.session_state.uploaded_file_path)\n elif st.session_state.chat_mode == \"image_processing\":\n if not processor.image_path and not processor.image_obj:\n response = \"šŸšØ Please provide an image first.\"\n else:\n action = st.session_state.image_action\n if action == \"Basic Detection\":\n response = processor.detect_image(query=user_input) or \"No result.\"\n elif action == \"Object Detection\":\n boxes = processor.detect_objects(prompt=user_input)\n response = f\"Detected boxes: {boxes}\" if boxes else \"No objects detected.\"\n elif action == \"Segmentation\":\n # Extract and return composite image for display\n segmented_image = processor.extract_segmentation_masks(\n output_dir=\"segmentation_results\",\n prompt=user_input\n )\n if segmented_image:\n # Add to chat history with an 'image' key\n st.session_state.chat_histories[st.session_state.chat_mode].append({\n \"role\": \"assistant\",\n \"content\": \"Segmentation complete.\",\n \"image\": segmented_image\n })\n response = \"Segmentation complete. Image displayed above.\"\n else:\n st.session_state.chat_histories[st.session_state.chat_mode].append({\n \"role\": \"assistant\",\n \"content\": \"Segmentation failed or no objects detected.\",\n \"image\": None\n })\n response = \"Segmentation failed or no objects detected.\"\n else:\n response = process_command(user_input)\n\n # --- Display response ---\n if isinstance(response, list) and isinstance(response[0], dict):\n for entry in response:\n with st.chat_message(\"assistant\"):\n st.markdown(f\"\"\"\n **šŸ”§ Function Executed:** `{entry.get('function_name', 'N/A')}` \n **šŸ“Œ Status:** āœ… `{entry.get('status', 'unknown')}` \n **šŸ“‚ Arguments:** `{entry.get('args', {})}` \n **šŸ“œ Output:** *{entry.get('output', 'No output.')}*\n \"\"\")\n add_message(\"assistant\", json.dumps(response, indent=2))\n else:\n st.chat_message(\"assistant\").markdown(response)\n add_message(\"assistant\", response)\n\n # --- Voice output ---\n if st.session_state.voice_output:\n full_resp = \"\\n\".join([r.get(\"output\",\"\") for r in response]) if isinstance(response,list) else str(response)\n audio_io = text_to_speech_local(full_resp.replace(\"*\",\"\"))\n st.markdown(f\"\"\"\n \n \"\"\", unsafe_allow_html=True)\n\n\n\n\n\n#------------------- Voice Input & Audio Handling -------------------\naudio_key = f\"audio_input_key_{st.session_state.audio_input_key_counter}\"\naudio_value = st.sidebar.audio_input(\"šŸŽ¤ Speak\", key=audio_key)\n\nif audio_value:\n with tempfile.TemporaryFile(suffix=\".wav\") as temp_audio:\n temp_audio.write(audio_value.getvalue())\n temp_audio.seek(0)\n transcribed_text = voice_to_text(temp_audio)\n\n if transcribed_text:\n st.chat_message(\"user\").markdown(transcribed_text)\n add_message(\"user\", transcribed_text)\n\n if st.session_state.chat_mode == \"chat_with_ai\":\n response = personal_chat_ai(transcribed_text)\n elif st.session_state.chat_mode == \"chat_with_rag\":\n subject = st.session_state.rag_subject\n response = chat_with_rag_session(subject, transcribed_text) if subject else \"šŸšØ Please enter a subject.\"\n elif st.session_state.chat_mode == \"data_analysis\":\n try:\n if not st.session_state.uploaded_file_path:\n response = \"šŸšØ Please upload a CSV file first.\"\n else:\n file_path = st.session_state.uploaded_file_path\n response = data_analysis(transcribed_text,file_path)\n except Exception as e:\n response = f\"Error reading CSV: {e}\"\n else:\n response = process_command(transcribed_text)\n \n if isinstance(response, list) and isinstance(response[0], dict):\n for entry in response:\n with st.chat_message(\"assistant\"):\n st.markdown(f\"\"\"\n **šŸ”§ Function Executed:** `{entry.get('function_name', 'N/A')}` \n **šŸ“Œ Status:** āœ… `{entry.get('status', 'unknown')}` \n **šŸ“‚ Arguments:** `{entry.get('args', {})}` \n **šŸ“œ Output:** *{entry.get('output', 'No output.')}*\n \"\"\")\n add_message(\"assistant\", json.dumps(response, indent=2))\n else:\n st.chat_message(\"assistant\").markdown(response)\n add_message(\"assistant\", response)\n\n # st.chat_message(\"assistant\").markdown(str(response))\n # add_message(\"assistant\", str(response))\n\n if st.session_state.voice_output:\n full_response = \"\\n\".join([sub_response.get(\"output\", \"\") for sub_response in response]) if isinstance(response, list) else str(response)\n audio_io = text_to_speech_local(full_response.replace(\"*\", \"\"))\n st.markdown(f\"\"\"\n \n \"\"\", unsafe_allow_html=True)\n\n del st.session_state[audio_key]\n st.session_state.audio_input_key_counter += 1\n\n\n# ----- Download History -----\nst.sidebar.download_button(\n label=\"šŸ“… Download Chat History\",\n data=json.dumps(st.session_state.chat_histories[st.session_state.chat_mode], indent=2),\n file_name=\"chat_history.json\",\n mime=\"application/json\"\n)\n" } ]