[
{
"path": ".gitattributes",
"content": "*.mp4 filter=lfs diff=lfs merge=lfs -text\nvideos/long_demo.mp4 filter=lfs diff=lfs merge=lfs -text\n"
},
{
"path": ".github/FUNDING.yml",
"content": "# These are supported funding model platforms\n\ngithub: # Replace with up to 4 GitHub Sponsors-enabled usernames e.g., [user1, user2]\npatreon: # Replace with a single Patreon username\nopen_collective: # Replace with a single Open Collective username\nko_fi: # Replace with a single Ko-fi username\ntidelift: # Replace with a single Tidelift platform-name/package-name e.g., npm/babel\ncommunity_bridge: # Replace with a single Community Bridge project-name e.g., cloud-foundry\nliberapay: # Replace with a single Liberapay username\nissuehunt: # Replace with a single IssueHunt username\nlfx_crowdfunding: # Replace with a single LFX Crowdfunding project-name e.g., cloud-foundry\npolar: # Replace with a single Polar username\nbuy_me_a_coffee: olney1\ncustom: ['https://ai-solutions.ai']\n"
},
{
"path": ".gitignore",
"content": "Credit to: https://djangowaves.com/tips-tricks/gitignore-for-a-django-project/\n\n.DS_Store\n\n# Ignore Pipfile and Pipfile.lock\nPipfile\nPipfile.lock\n\n\n# Django #\n*.log\n*.pot\n*.pyc\n__pycache__\ndb.sqlite3\n\n# Backup files # \n*.bak \n\n# If you are using PyCharm # \n# User-specific stuff\n.idea/**/workspace.xml\n.idea/**/tasks.xml\n.idea/**/usage.statistics.xml\n.idea/**/dictionaries\n.idea/**/shelf\n\n# AWS User-specific\n.idea/**/aws.xml\n\n# Generated files\n.idea/**/contentModel.xml\n\n# Sensitive or high-churn files\n.idea/**/dataSources/\n.idea/**/dataSources.ids\n.idea/**/dataSources.local.xml\n.idea/**/sqlDataSources.xml\n.idea/**/dynamic.xml\n.idea/**/uiDesigner.xml\n.idea/**/dbnavigator.xml\n\n# Gradle\n.idea/**/gradle.xml\n.idea/**/libraries\n\n# File-based project format\n*.iws\n\n# IntelliJ\nout/\n\n# JIRA plugin\natlassian-ide-plugin.xml\n\n# Python # \n*.py[cod] \n*$py.class \n\n# Distribution / packaging \n.Python build/ \ndevelop-eggs/ \ndist/ \ndownloads/ \neggs/ \n.eggs/ \nlib/ \nlib64/ \nparts/ \nsdist/ \nvar/ \nwheels/ \n*.egg-info/ \n.installed.cfg \n*.egg \n*.manifest \n*.spec \n\n# Installer logs \npip-log.txt \npip-delete-this-directory.txt \n\n# Unit test / coverage reports \nhtmlcov/ \n.tox/ \n.coverage \n.coverage.* \n.cache \n.pytest_cache/ \nnosetests.xml \ncoverage.xml \n*.cover \n.hypothesis/ \n\n# Jupyter Notebook \n.ipynb_checkpoints \n\n# pyenv \n.python-version \n\n# celery \ncelerybeat-schedule.* \n\n# SageMath parsed files \n*.sage.py \n\n# Environments \n.env \n.venv \nenv/ \nvenv/ \nENV/ \nenv.bak/ \nvenv.bak/ \n\n# mkdocs documentation \n/site \n\n# mypy \n.mypy_cache/ \n\n# Sublime Text # \n*.tmlanguage.cache \n*.tmPreferences.cache \n*.stTheme.cache \n*.sublime-workspace \n*.sublime-project \n\n# sftp configuration file \nsftp-config.json \n\n# Package control specific files Package \nControl.last-run \nControl.ca-list \nControl.ca-bundle \nControl.system-ca-bundle \nGitHub.sublime-settings \n\n# Visual Studio Code # \n.vscode/* \n!.vscode/settings.json \n!.vscode/tasks.json \n!.vscode/launch.json \n!.vscode/extensions.json \n.history\n\n# Additional Removals\nresponse.mp3\ntest.py"
},
{
"path": "LICENSE",
"content": "MIT License\n\nCopyright (c) 2023 Ben\n\nPermission is hereby granted, free of charge, to any person obtaining a copy\nof this software and associated documentation files (the \"Software\"), to deal\nin the Software without restriction, including without limitation the rights\nto use, copy, modify, merge, publish, distribute, sublicense, and/or sell\ncopies of the Software, and to permit persons to whom the Software is\nfurnished to do so, subject to the following conditions:\n\nThe above copyright notice and this permission notice shall be included in all\ncopies or substantial portions of the Software.\n\nTHE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\nIMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\nFITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\nAUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\nLIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\nOUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\nSOFTWARE.\n"
},
{
"path": "README.md",
"content": "# ChatGPT Smart Speaker (speech recognition and text-to-speech using OpenAI and Google Speech Recognition)\n\n\n\n\n\n## Video Demo using activation word \"Jeffers\" - [Demo](https://vimeo.com/1029160996?share=copy#t=0)\n
\n
\n\n## Equipment List:\n\n## - [Raspberry Pi 4b 4GB](https://www.amazon.co.uk/Raspberry-Pi-Model-4GB/dp/B09TTNF8BT?_encoding=UTF8&tag=olney104-21 \"Raspberry Pi 4b 4GB\")\n## - [VMini External USB Stereo Speaker](https://www.amazon.co.uk/Speakers-Computer-Speaker-Soundbar-Checkout/dp/B08NDJDFPS?_encoding=UTF8&tag=olney104-21 \"VMini External USB Stereo Speaker\")\n## - [VReSpeaker 4-Mic Array](https://www.amazon.co.uk/Seeed-ReSpeaker-4-Mic-Array-Raspberry/dp/B076SSR1W1?&_encoding=UTF8&tag=olney104-21 \"VReSpeaker 4-Mic Array\")\n## - [ANSMANN 10,000mAh Type-C 20W PD Power Bank](https://www.amazon.co.uk/Powerbank-10000mAh-capacity-Smartphones-rechargeable-Black/dp/B01NBNH2AL/?_encoding=UTF8&tag=olney104-21 \"ANSMANN 10,000mAh Type-C 20W PD Power Bank\")\n\n
\n\n## Running on your PC/Mac (use the chat.py or test.py script)\n\nThe `chat.py` and `test.py` scripts run directly on your PC/Mac. They both allow you to use speech recognition to input a prompt, send the prompt to OpenAI to generate a response, and then use gTTS to convert the response to an audio file and play the audio file on your Mac/PC. Your PC/Mac must have a working default microphone and speakers for this script to work. Please note that these scripts were designed on a Mac, so additional dependencies may be required on Windows and Linux. The difference between them is that `chat.py` is faster and always on and `test.py` acts like a standard smart speaker - only working once it hears the activation command (currently set to 'Jeffers').\n\n
\n\n## Running on Raspberry Pi (use the pi.py script)\n\n\n The `pi.py` script is a new and more advanced custom version of the `smart_speaker.py` script and is the most advanced script similar to a real smart speaker. The purpose of this script is to offload the wake up word to a custom model build via PicoVoice (`https://console.picovoice.ai/`). This improves efficiency and long term usage reliability. This script will be the main script for development moving forward due to greater reliability and more advanced features to be added regularly.\n\n
\n\n## Prerequisites - chat.py\n\n- You need to have a valid OpenAI API key. You can sign up for a free API key at https://platform.openai.com.\n- You'll need to be running Python version 3.7.3 or higher. I am using 3.11.4 on a Mac and 3.7.3 on Raspberry Pi.\n- Run `brew install portaudio` after installing HomeBrew: `/bin/bash -c \"$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)\"`\n- You need to install the following packages: `openai`, `gTTS`, `pyaudio`, `SpeechRecognition`, `playsound, python-dotenv` and `pyobjc` if you are on a Mac. You can install these packages using pip or use pipenv if you wish to contain a virtual environment. \n- Firstly, update your tools: `pip install --upgrade pip setuptools` then `pip install openai pyaudio SpeechRecognition gTTS playsound python-dotenv apa102-pi gpiozero pyobjc`\n\n
\n\n## Prerequisites - pi.py \nTo run pi.py you will need a Raspberry Pi 4b (I'm using the 4GB model but 2GB should be enough), ReSpeaker 4-Mic Array for Raspberry Pi and USB speakers.\n\nYou will also need a developer account and API key with OpenAI (`https://platform.openai.com/overview`), a Tavily Search agent API key (`https://app.tavily.com/sign-in`) and an Access Key and Custom Voice Model with PicoVoice (`https://console.picovoice.ai/`) and (`https://console.picovoice.ai/ppn` respectively. Please create your own voice model and download the correct version for use on a Raspberry Pi)\n\nNow on to the Pi setup. Let's get started!\n\nRun the following on your Raspberry Pi terminal:\n\n1. `sudo apt update`\n\n2. `sudo apt install python3-gpiozero`\n\n3. `git clone https://github.com/Olney1/ChatGPT-OpenAI-Smart-Speaker`\n\n4. Firstly, update your tools: `pip install --upgrade pip setuptools` then `pip install openai pyaudio SpeechRecognition gTTS pydub python-dotenv apa102-pi gpiozero` Next, install the dependencies, `pip install -r requirements.txt`. I am using Python 3.9 `#!/usr/bin/env python3.9`. You can install these packages using pip or use pipenv if you wish to contain a virtual environment.\n\n5. PyAudio relies on PortAudio as a dependency. You can install it using the following command: `sudo apt-get install portaudio19-dev`\n\n6. Pydub dependencies: You need to have ffmpeg installed on your system. On a Raspberry Pi you can install it using: `sudo apt-get install ffmpeg`. You may also need simpleaudio if you run into issues with the script hanging when finding the wake word, so it's best to install these packages just in case: `sudo apt-get install python3-dev` (for development headers to compile) and `install simpleaudio` (for a different backend to play mp3 files) and `sudo apt-get install libasound2-dev` (necessary dependencies).\n\n7. If you are using the RESPEAKER, follow this guide to install the required dependencies: (`https://wiki.seeedstudio.com/ReSpeaker_4_Mic_Array_for_Raspberry_Pi/#getting-started`). Then install support for the lights on the RESPEAKER board. You'll need APA102 LED: `sudo apt install -y python3-rpi.gpio` and then `sudo pip3 install apa102-pi`.\n\n8. Activate SPI: sudo raspi-config; Go to \"Interface Options\"; Go to \"SPI\"; Enable SPI; While you are at it: Do change the default password! Exit the tool and reboot.\n\n9. Get the Seeed voice card source code, install and reboot: \n`git clone https://github.com/HinTak/seeed-voicecard.git`\n`cd seeed-voicecard`\n`sudo ./install.sh`\n`sudo reboot now`\n\n10. Finally, load audio output on Raspberry Pi `sudo raspi-config`\n-Select 1 System options\n-Select S2 Audio\n-Select your preferred Audio output device\n-Select Finish\n\n
\n\n## Usage - applies to chat.py:\n\n1. You'll need to set up the environment variables for your Open API Key. To do this create a `.env` file in the same directory and add your API Key to the file like this: `OPENAI_API_KEY=\"API KEY GOES HERE\"`. This is safer than hard coding your API key into the program.\nYou must not change the name of the variable `OPENAI_API_KEY`.\n2. Run the script using `python chat.py`.\n3. The script will prompt you to say something. Speak a sentence into your microphone. You may need to allow the program permission to access your microphone on a Mac, a prompt should appear when running the program.\n4. The script will send the spoken sentence to OpenAI, generate a response using the text-to-speech model, and play the response as an audio file.\n\n
\n\n## Usage - applies to pi.py\n1. You'll need to set up the environment variables for your Open API Key, PicoVoice Access Key and Tavily API key for agent searches. To do this create a `.env` file in the same directory and add your API Keys to the file like this: `OPENAI_API_KEY=\"API KEY GOES HERE\"` and `ACCESS_KEY=\"PICOVOICE ACCESS KEY GOES HERE\"` and `TAVILY_API_KEY=\"API KEY GOES HERE\"`. This is safer than hard coding your API key into the program.\n2. Ensure that you have the `pi.py` script along with `apa102.py` and `alexa_led_pattern.py` scripts in the same folder saved on your Pi if using ReSpeaker.\n3. Run the script using `python3 pi.py` or `python3 pi.py 2> /dev/null` on the Raspberry Pi. The second option omits all developer warnings and errors to keep the console focused purely on the print statements.\n4. The script will prompt you to say the wake word which is programmed into the wake word custom model by Picovoice as 'Jeffers'. You can change this to any name you want. Once the wake word has been detected the lights will light up blue. It will now be ready for you to ask your question. When you have asked your question, or when the microphone picks up and processes noise, the lights will rotate a blue colour meaning that your recording sample/question is being sent to OpenAI.\n5. The script will then generate a response using the text-to-speech model, and play the response as an audio file.\n\n## Customisation\n\n- You can change the OpenAI model engine by modifying the value of `model_engine`. For example, to use the \"gpt-3.5-turbo\" model for a cheaper and quicker response but with a knowledge cut-off to Sep 2021, set `model_engine = \"gpt-3.5-turbo\"`.\n- You can change the language of the generated audio file by modifying the value of `language`. For example, to generate audio in French, set `language = 'fr'`.\n- You can adjust the `temperature` parameter in the following line to control the randomness of the generated response:\n\n```\nresponse = client.chat.completions.create(\n model=model_engine,\n messages=[{\"role\": \"system\", \"content\": \"You are a helpful smart speaker called Jeffers!\"}, # Play about with more context here.\n {\"role\": \"user\", \"content\": prompt}],\n max_tokens=1024,\n n=1,\n temperature=0.7,\n )\n return response\n```\n\nHigher values of `temperature` will result in more diverse and random responses, while lower values will result in more deterministic responses.\n\n
\n\n## Important notes for Raspberry Pi Installation\n\nAs of May 2024, Seeed Studio has listed the ReSpeaker series among its [retired products](https://wiki.seeedstudio.com/discontinuedproducts/). It may not be compatible with the Raspberry Pi 5 due to hardware changes.\n\nIt is highly recommended to install the legacy version of Raspberry Pi on a Rasberry Pi 4b model if you have an ReSPEAKER. You can also simply buy a micro USB microphone and configure the input source for this using alsamixer and currently still use the ReSPEAKER for the lighting pattern.\n\nIf you are using the same USB speaker in my video you will need to run `sudo apt-get install pulseaudio` to install support for this. This may also require you to set a command to start pulseaudio on every boot: `pulseaudio --start`.\n\n### Adding a Start Command on Boot\n\nOpen the terminal and type: `sudo nano /etc/rc. local`\n\nAfter important network/start commands add this: `su -l pi -c '/usr/bin/python3 /home/pi/ChatGPT-OpenAI-Smart-Speaker/ && pulseaudio --start && python3 pi.py 2> /dev/null’`\n\nBe sure to leave the line exit 0 at the end, then save the file and exit. In nano, to exit, type Ctrl-x, and then Y\n\n### ReSpeaker\n\nIf you want to use ReSpeaker for the lights, you can purchase this from most of the major online stores that stock Raspberry Pi. \nHere is the online guide: https://wiki.seeedstudio.com/ReSpeaker_4_Mic_Array_for_Raspberry_Pi/\n\nTo test your microphone and speakers install Audacity on your Raspberry Pi:\n\n`sudo apt update`\n\n`sudo apt install audacity`\n\n`audacity`\n\n### Other Possible Issues\n\nOn the raspberry pi you may encounter an error regarding the installation of `flac`.\n\nSee here for the resolution: https://raspberrypi.stackexchange.com/questions/137630/im-unable-to-install-flac-on-my-raspberry-pi-3\n\nThe files you will need are going to be here: https://archive.raspbian.org/raspbian/pool/main/f/flac/\n
Please note the links below may have changed or be updated, so please refer back to this link above for the latest file names and then update your command below.\n \n`sudo apt-get install libogg0`\n\n`$ wget https://archive.raspbian.org/raspbian/pool/main/f/flac/libflac8_1.3.2-3+deb10u3_armhf.deb`\n\n`$ wget https://archive.raspbian.org/raspbian/pool/main/f/flac/flac_1.3.2-3+deb10u3_armhf.deb`\n\n`$ sudo dpkg -i libflac8_1.3.2-3+deb10u3_armhf.deb` \n\n`$ sudo dpkg -i flac_1.3.2-3+deb10u3_armhf.deb`\n\n`$ which flac`\n`/usr/bin/flac`\n\n`sudo reboot`\n\n`$ flac --version`\n`flac 1.3.2`\n\nYou may find you need to install GStreamer if you encounter errors regarding Gst.\n\nInstall GStreamer: Open a terminal and run the following command to install GStreamer and its base plugins:\n\n`sudo apt-get install gstreamer1.0-tools gstreamer1.0-plugins-base gstreamer1.0-plugins-good`\nThis installs the GStreamer core, along with a set of essential and good-quality plugins.\n\nNext, you need to install the Python bindings for GStreamer. Use this command:\n\n`sudo apt-get install python3-gst-1.0`\nThis command installs the GStreamer bindings for Python 3.\n\nInstall Additional GStreamer Plugins (if needed): Depending on the audio formats you need to work with, you might need additional GStreamer plugins. For example, to install plugins for MP3 playback, use:\n\n`sudo apt-get install gstreamer1.0-plugins-ugly`\n\nTo quit a running script on Pi from boot: `ALT + PrtScSysRq (or Print button) + K`\n\n
\n\n## Credit to:\nhttps://github.com/tinue/apa102-pi & Seeed Technology Limited for supplementary code.\n\n
\n\n## Read more about what is next for the project\nhttps://medium.com/@ben_olney/openai-smart-speaker-with-raspberry-pi-5e284d21a53e"
},
{
"path": "alexa_led_pattern.py",
"content": "#!/usr/bin/env python\n\n# Copyright (C) 2017 Seeed Technology Limited\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nimport time\n\nclass AlexaLedPattern(object):\n def __init__(self, show=None, number=12):\n self.pixels_number = number\n self.pixels = [0] * 4 * number\n\n if not show or not callable(show):\n def dummy(data):\n pass\n show = dummy\n\n self.show = show\n self.stop = False\n\n def wakeup(self, direction=0):\n position = int((direction + 15) / (360 / self.pixels_number)) % self.pixels_number\n\n pixels = [0, 0, 0, 24] * self.pixels_number\n pixels[position * 4 + 2] = 48\n\n self.show(pixels)\n\n def listen(self):\n pixels = [0, 0, 0, 24] * self.pixels_number\n\n self.show(pixels)\n\n def think(self):\n pixels = [0, 0, 12, 12, 0, 0, 0, 24] * self.pixels_number\n\n while not self.stop:\n self.show(pixels)\n time.sleep(0.2)\n pixels = pixels[-4:] + pixels[:-4]\n\n def speak(self):\n step = 1\n position = 12\n while not self.stop:\n pixels = [0, 0, position, 24 - position] * self.pixels_number\n self.show(pixels)\n time.sleep(0.01)\n if position <= 0:\n step = 1\n time.sleep(0.4)\n elif position >= 12:\n step = -1\n time.sleep(0.4)\n\n position += step\n\n def off(self):\n self.show([0] * 4 * 12)\n"
},
{
"path": "apa102.py",
"content": "\"\"\"\nfrom https://github.com/tinue/APA102_Pi\nThis is the main driver module for APA102 LEDs\n\"\"\"\nimport spidev\nfrom math import ceil\n\nRGB_MAP = { 'rgb': [3, 2, 1], 'rbg': [3, 1, 2], 'grb': [2, 3, 1],\n 'gbr': [2, 1, 3], 'brg': [1, 3, 2], 'bgr': [1, 2, 3] }\n\nclass APA102:\n \"\"\"\n Driver for APA102 LEDS (aka \"DotStar\").\n\n (c) Martin Erzberger 2016-2017\n\n My very first Python code, so I am sure there is a lot to be optimized ;)\n\n Public methods are:\n - set_pixel\n - set_pixel_rgb\n - show\n - clear_strip\n - cleanup\n\n Helper methods for color manipulation are:\n - combine_color\n - wheel\n\n The rest of the methods are used internally and should not be used by the\n user of the library.\n\n Very brief overview of APA102: An APA102 LED is addressed with SPI. The bits\n are shifted in one by one, starting with the least significant bit.\n\n An LED usually just forwards everything that is sent to its data-in to\n data-out. While doing this, it remembers its own color and keeps glowing\n with that color as long as there is power.\n\n An LED can be switched to not forward the data, but instead use the data\n to change it's own color. This is done by sending (at least) 32 bits of\n zeroes to data-in. The LED then accepts the next correct 32 bit LED\n frame (with color information) as its new color setting.\n\n After having received the 32 bit color frame, the LED changes color,\n and then resumes to just copying data-in to data-out.\n\n The really clever bit is this: While receiving the 32 bit LED frame,\n the LED sends zeroes on its data-out line. Because a color frame is\n 32 bits, the LED sends 32 bits of zeroes to the next LED.\n As we have seen above, this means that the next LED is now ready\n to accept a color frame and update its color.\n\n So that's really the entire protocol:\n - Start by sending 32 bits of zeroes. This prepares LED 1 to update\n its color.\n - Send color information one by one, starting with the color for LED 1,\n then LED 2 etc.\n - Finish off by cycling the clock line a few times to get all data\n to the very last LED on the strip\n\n The last step is necessary, because each LED delays forwarding the data\n a bit. Imagine ten people in a row. When you yell the last color\n information, i.e. the one for person ten, to the first person in\n the line, then you are not finished yet. Person one has to turn around\n and yell it to person 2, and so on. So it takes ten additional \"dummy\"\n cycles until person ten knows the color. When you look closer,\n you will see that not even person 9 knows its own color yet. This\n information is still with person 2. Essentially the driver sends additional\n zeroes to LED 1 as long as it takes for the last color frame to make it\n down the line to the last LED.\n \"\"\"\n # Constants\n MAX_BRIGHTNESS = 31 # Safeguard: Set to a value appropriate for your setup\n LED_START = 0b11100000 # Three \"1\" bits, followed by 5 brightness bits\n\n def __init__(self, num_led, global_brightness=MAX_BRIGHTNESS,\n order='rgb', bus=0, device=1, max_speed_hz=8000000):\n self.num_led = num_led # The number of LEDs in the Strip\n order = order.lower()\n self.rgb = RGB_MAP.get(order, RGB_MAP['rgb'])\n # Limit the brightness to the maximum if it's set higher\n if global_brightness > self.MAX_BRIGHTNESS:\n self.global_brightness = self.MAX_BRIGHTNESS\n else:\n self.global_brightness = global_brightness\n\n self.leds = [self.LED_START,0,0,0] * self.num_led # Pixel buffer\n self.spi = spidev.SpiDev() # Init the SPI device\n self.spi.open(bus, device) # Open SPI port 0, slave device (CS) 1\n # Up the speed a bit, so that the LEDs are painted faster\n if max_speed_hz:\n self.spi.max_speed_hz = max_speed_hz\n\n def clock_start_frame(self):\n \"\"\"Sends a start frame to the LED strip.\n\n This method clocks out a start frame, telling the receiving LED\n that it must update its own color now.\n \"\"\"\n self.spi.xfer2([0] * 4) # Start frame, 32 zero bits\n\n\n def clock_end_frame(self):\n \"\"\"Sends an end frame to the LED strip.\n\n As explained above, dummy data must be sent after the last real colour\n information so that all of the data can reach its destination down the line.\n The delay is not as bad as with the human example above.\n It is only 1/2 bit per LED. This is because the SPI clock line\n needs to be inverted.\n\n Say a bit is ready on the SPI data line. The sender communicates\n this by toggling the clock line. The bit is read by the LED\n and immediately forwarded to the output data line. When the clock goes\n down again on the input side, the LED will toggle the clock up\n on the output to tell the next LED that the bit is ready.\n\n After one LED the clock is inverted, and after two LEDs it is in sync\n again, but one cycle behind. Therefore, for every two LEDs, one bit\n of delay gets accumulated. For 300 LEDs, 150 additional bits must be fed to\n the input of LED one so that the data can reach the last LED.\n\n Ultimately, we need to send additional numLEDs/2 arbitrary data bits,\n in order to trigger numLEDs/2 additional clock changes. This driver\n sends zeroes, which has the benefit of getting LED one partially or\n fully ready for the next update to the strip. An optimized version\n of the driver could omit the \"clockStartFrame\" method if enough zeroes have\n been sent as part of \"clockEndFrame\".\n \"\"\"\n # Round up num_led/2 bits (or num_led/16 bytes)\n for _ in range((self.num_led + 15) // 16):\n self.spi.xfer2([0x00])\n\n\n def clear_strip(self):\n \"\"\" Turns off the strip and shows the result right away.\"\"\"\n\n for led in range(self.num_led):\n self.set_pixel(led, 0, 0, 0)\n self.show()\n\n\n def set_pixel(self, led_num, red, green, blue, bright_percent=100):\n \"\"\"Sets the color of one pixel in the LED stripe.\n\n The changed pixel is not shown yet on the Stripe, it is only\n written to the pixel buffer. Colors are passed individually.\n If brightness is not set the global brightness setting is used.\n \"\"\"\n if led_num < 0:\n return # Pixel is invisible, so ignore\n if led_num >= self.num_led:\n return # again, invisible\n\n # Calculate pixel brightness as a percentage of the\n # defined global_brightness. Round up to nearest integer\n # as we expect some brightness unless set to 0\n brightness = ceil(bright_percent*self.global_brightness/100.0)\n brightness = int(brightness)\n\n # LED startframe is three \"1\" bits, followed by 5 brightness bits\n ledstart = (brightness & 0b00011111) | self.LED_START\n\n start_index = 4 * led_num\n self.leds[start_index] = ledstart\n self.leds[start_index + self.rgb[0]] = red\n self.leds[start_index + self.rgb[1]] = green\n self.leds[start_index + self.rgb[2]] = blue\n\n\n def set_pixel_rgb(self, led_num, rgb_color, bright_percent=100):\n \"\"\"Sets the color of one pixel in the LED stripe.\n\n The changed pixel is not shown yet on the Stripe, it is only\n written to the pixel buffer.\n Colors are passed combined (3 bytes concatenated)\n If brightness is not set the global brightness setting is used.\n \"\"\"\n self.set_pixel(led_num, (rgb_color & 0xFF0000) >> 16,\n (rgb_color & 0x00FF00) >> 8, rgb_color & 0x0000FF,\n bright_percent)\n\n\n def rotate(self, positions=1):\n \"\"\" Rotate the LEDs by the specified number of positions.\n\n Treating the internal LED array as a circular buffer, rotate it by\n the specified number of positions. The number could be negative,\n which means rotating in the opposite direction.\n \"\"\"\n cutoff = 4 * (positions % self.num_led)\n self.leds = self.leds[cutoff:] + self.leds[:cutoff]\n\n\n def show(self):\n \"\"\"Sends the content of the pixel buffer to the strip.\n\n Todo: More than 1024 LEDs requires more than one xfer operation.\n \"\"\"\n self.clock_start_frame()\n # xfer2 kills the list, unfortunately. So it must be copied first\n # SPI takes up to 4096 Integers. So we are fine for up to 1024 LEDs.\n self.spi.xfer2(list(self.leds))\n self.clock_end_frame()\n\n\n def cleanup(self):\n \"\"\"Release the SPI device; Call this method at the end\"\"\"\n\n self.spi.close() # Close SPI port\n\n @staticmethod\n def combine_color(red, green, blue):\n \"\"\"Make one 3*8 byte color value.\"\"\"\n\n return (red << 16) + (green << 8) + blue\n\n\n def wheel(self, wheel_pos):\n \"\"\"Get a color from a color wheel; Green -> Red -> Blue -> Green\"\"\"\n\n if wheel_pos > 255:\n wheel_pos = 255 # Safeguard\n if wheel_pos < 85: # Green -> Red\n return self.combine_color(wheel_pos * 3, 255 - wheel_pos * 3, 0)\n if wheel_pos < 170: # Red -> Blue\n wheel_pos -= 85\n return self.combine_color(255 - wheel_pos * 3, 0, wheel_pos * 3)\n # Blue -> Green\n wheel_pos -= 170\n return self.combine_color(0, wheel_pos * 3, 255 - wheel_pos * 3)\n\n\n def dump_array(self):\n \"\"\"For debug purposes: Dump the LED array onto the console.\"\"\"\n\n print(self.leds)\n"
},
{
"path": "chat.py",
"content": "from openai import OpenAI\nimport os\nimport speech_recognition as sr\nfrom gtts import gTTS\nfrom playsound import playsound\nfrom dotenv import load_dotenv\nfrom pathlib import Path\n\n# Load the environment variables\nload_dotenv()\n\n# Create an OpenAI API client\nclient = OpenAI(api_key=os.environ.get(\"OPENAI_API_KEY\"))\n# Model name and language\nmodel_engine = \"gpt-4o\"\nlanguage = 'en'\n\ndef recognise_speech():\n # obtain audio from the microphone\n r = sr.recogniser()\n with sr.Microphone() as source:\n print(\"Say something!\")\n audio = r.listen(source)\n\n # recognise speech using Google Speech Recognition\n try:\n # for testing purposes, we're just using the default API key\n # to use another API key, use `r.recognise_google(audio, key=\"GOOGLE_SPEECH_RECOGNITION_API_KEY\")`\n # instead of `r.recognise_google(audio)`\n # convert the audio to text\n print(\"Google Speech Recognition thinks you said: \" + r.recognise_google(audio))\n speech = r.recognise_google(audio)\n print(\"This is what we think was said: \" + speech)\n except sr.UnknownValueError:\n print(\"Google Speech Recognition could not understand audio\")\n except sr.RequestError as e:\n print(\"Could not request results from Google Speech Recognition service; {0}\".format(e))\n\n # Add a holding messsage like the one below to deal with current TTS delays until such time that TTS can be streamed.\n playsound(\"sounds/holding.mp3\") # There’s an optional second argument, block, which is set to True by default. Setting it to False makes the function run asynchronously.\n\n return speech\n\ndef chatgpt_response(prompt):\n # send the converted audio text to chatgpt\n response = client.chat.completions.create(\n model=model_engine,\n messages=[{\"role\": \"system\", \"content\": \"You are a helpful smart speaker called Jeffers!\"},\n {\"role\": \"user\", \"content\": prompt}],\n max_tokens=300,\n n=1,\n temperature=0.7,\n )\n return response\n\ndef generate_audio_file(message):\n speech_file_path = Path(__file__).parent / \"response.mp3\"\n response = client.audio.speech.create(\n model=\"tts-1\",\n voice=\"fable\",\n input=message\n )\n # response.content contains the binary audio data which we can write to a file and play\n with open(speech_file_path, 'wb') as f:\n f.write(response.content)\n\ndef play_audio_file():\n # play the audio file\n playsound(\"response.mp3\") # There’s an optional second argument, block, which is set to True by default. Setting it to False makes the function run asynchronously.\n\ndef main():\n # run the program\n prompt = recognise_speech()\n print(f\"This is the prompt being sent to OpenAI: \" + prompt)\n responses = chatgpt_response(prompt)\n message = responses.choices[0].message.content\n print(message)\n generate_audio_file(message)\n play_audio_file()\n\nif __name__ == \"__main__\":\n main()"
},
{
"path": "create_messages.py",
"content": "from openai import OpenAI\nimport os\nfrom dotenv import load_dotenv\n\n\"\"\"Create your own professional messages with OpenAI for your speaker\"\"\"\n\n# Load the environment variables\nload_dotenv()\n\nclient = OpenAI(api_key=os.environ.get(\"OPENAI_API_KEY\"))\n\ndef create_holding_message():\n\n message = \"One moment please\"\n\n response = client.audio.speech.create(\n model=\"tts-1\",\n voice=\"fable\",\n input=message,\n )\n\n response.stream_to_file(\"sounds/holding.mp3\")\n\n\n\ndef create_google_speech_issue():\n\n message = \"Sorry, there was an issue reaching Google Speech Recognition, please try again.\"\n\n response = client.audio.speech.create(\n model=\"tts-1\",\n voice=\"fable\",\n input=message,\n )\n\n response.stream_to_file(\"sounds/google_issue.mp3\")\n\n\ndef understand_speech_issue():\n\n message = \"Sorry, I didn't quite get that.\"\n\n response = client.audio.speech.create(\n model=\"tts-1\",\n voice=\"fable\",\n input=message,\n )\n\n response.stream_to_file(\"sounds/understand.mp3\")\n\n\ndef stop():\n\n message = \"No worries, I'll be here when you need me.\"\n\n response = client.audio.speech.create(\n model=\"tts-1\",\n voice=\"fable\",\n input=message,\n )\n\n response.stream_to_file(\"sounds/stop.mp3\")\n\n\ndef hello():\n\n message = \"Welcome, my name is Jeffers, I'm your helpful smart speaker. Just say my name and ask me anything.\"\n\n response = client.audio.speech.create(\n model=\"tts-1\",\n voice=\"fable\",\n input=message,\n )\n\n response.stream_to_file(\"sounds/hello.mp3\")\n\n\ndef create_picovoice_issue():\n\n message = \"Sorry, there was an issue with the PicoVoice Service.\"\n\n response = client.audio.speech.create(\n model=\"tts-1\",\n voice=\"fable\",\n input=message,\n )\n\n response.stream_to_file(\"sounds/picovoice_issue.mp3\")\n\n\ndef create_picture_message():\n\n message = \"Let me take a look through the camera.\"\n\n response = client.audio.speech.create(\n model=\"tts-1\",\n voice=\"fable\",\n input=message,\n )\n\n response.stream_to_file(\"sounds/start_camera.mp3\")\n\n\ndef start_picture_message():\n\n message = \"Hold steady....... I'm taking a photo now...... in ....... 3 ...... 2 ......... 1\"\n\n response = client.audio.speech.create(\n model=\"tts-1\",\n voice=\"fable\",\n input=message,\n )\n\n response.stream_to_file(\"sounds/take_photo.mp3\")\n\n\ndef agent_search():\n\n message = \"Let me do a quick search for you.\"\n\n response = client.audio.speech.create(\n model=\"tts-1\",\n voice=\"fable\",\n input=message,\n )\n\n response.stream_to_file(\"sounds/agent.mp3\")\n\ndef audio_issue():\n\n message = \"There was an issue opening the PyAudio stream on the device.\"\n\n response = client.audio.speech.create(\n model=\"tts-1\",\n voice=\"fable\",\n input=message,\n )\n\n response.stream_to_file(\"sounds/audio_issue.mp3\")\n\ndef tavily_key_error():\n \n message = \"I could not find your API key for the Tavily Search Service. Please ensure you update your .env file with a Tavily Search API key in order to use the agent.\"\n \n response = client.audio.speech.create(\n model=\"tts-1\",\n voice=\"fable\",\n input=message,\n )\n \n response.stream_to_file(\"sounds/tavily_key_error.mp3\")\n\ndef camera_issue():\n\n message = \"Sorry, there was an issue opening Pi Camera.\"\n\n response = client.audio.speech.create(\n model=\"tts-1\",\n voice=\"fable\",\n input=message,\n )\n\n response.stream_to_file(\"sounds/camera_issue.mp3\")\n\ncamera_issue()"
},
{
"path": "deprecated/smart_speaker.py",
"content": "import os\nfrom openai import OpenAI\nimport pyaudio\nimport speech_recognition as sr\nfrom gtts import gTTS\nfrom dotenv import load_dotenv\nimport apa102\nimport threading\nfrom gpiozero import LED\ntry:\n import queue as Queue\nexcept ImportError:\n import Queue as Queue\nfrom alexa_led_pattern import AlexaLedPattern\nfrom pathlib import Path\nfrom pydub import AudioSegment\nfrom pydub.playback import play\nimport time\n\n# Set the working directory for Pi if you want to run this code via rc.local script so that it is automatically running on Pi startup. Remove this line if you have installed this project in a different directory.\nos.chdir('/home/pi/ChatGPT-OpenAI-Smart-Speaker')\n\n# Set the pre-prompt configuration here to precede the user's question to enable OpenAI to understand that it's acting as a smart speaker and add any other required information. We will send this in the OpenAI call as part of the system content in messages.\npre_prompt = \"You are a helpful smart speaker called Jeffers! Please respond with short and concise answers to the following user question and always remind the user at the end to say your name again to continue the conversation:\"\n \n# Load the environment variables\nload_dotenv()\n# Create an OpenAI API client\nclient = OpenAI(api_key=os.environ.get(\"OPENAI_API_KEY\"))\n\n# Add 1 second silence globally due to initial buffering how pydub handles audio in memory\nsilence = AudioSegment.silent(duration=1000) \n \n# load pixels Class\nclass Pixels:\n PIXELS_N = 12\n \n def __init__(self, pattern=AlexaLedPattern):\n self.pattern = pattern(show=self.show)\n self.dev = apa102.APA102(num_led=self.PIXELS_N)\n self.power = LED(5)\n self.power.on()\n self.queue = Queue.Queue()\n self.thread = threading.Thread(target=self._run)\n self.thread.daemon = True\n self.thread.start()\n self.last_direction = None\n \n def wakeup(self, direction=0):\n self.last_direction = direction\n def f():\n self.pattern.wakeup(direction)\n \n self.put(f)\n \n def listen(self):\n if self.last_direction:\n def f():\n self.pattern.wakeup(self.last_direction)\n self.put(f)\n else:\n self.put(self.pattern.listen)\n \n def think(self):\n self.put(self.pattern.think)\n \n def speak(self):\n self.put(self.pattern.speak)\n \n def off(self):\n self.put(self.pattern.off)\n \n def put(self, func):\n self.pattern.stop = True\n self.queue.put(func)\n \n def _run(self):\n while True:\n func = self.queue.get()\n self.pattern.stop = False\n func()\n \n def show(self, data):\n for i in range(self.PIXELS_N):\n self.dev.set_pixel(i, int(data[4*i + 1]), int(data[4*i + 2]), int(data[4*i + 3]))\n \n self.dev.show()\n \npixels = Pixels()\n \n \n# settings and keys\nmodel_engine = \"gpt-4o\"\nlanguage = 'en'\n \ndef recognise_speech():\n # obtain audio from the microphone\n r = sr.Recognizer()\n with sr.Microphone() as source:\n try:\n pixels.off()\n print(\"Listening...\")\n audio_stream = r.listen(source)\n print(\"Waiting for wake word...\")\n # recognize speech using Google Speech Recognition\n try:\n # convert the audio to text\n print(\"Google Speech Recognition thinks you said \" + r.recognize_google(audio_stream))\n speech = r.recognize_google(audio_stream)\n print(\"Recognized Speech:\", speech) # Print the recognized speech for debugging\n words = speech.lower().split() # Split the speech into words\n if \"jeffers\" not in words:\n print(\"Wake word not detected in the speech\")\n return False\n else:\n print(\"Found wake word!\")\n # Add 1 second silence due to initial buffering how pydub handles audio in memory\n silence = AudioSegment.silent(duration=1000) \n start_audio_response = silence + AudioSegment.from_mp3(\"sounds/start.mp3\")\n play(start_audio_response)\n return True\n except sr.UnknownValueError:\n print(\"Google Speech Recognition could not understand audio\")\n except sr.RequestError as e:\n print(\"Could not request results from Google Speech Recognition service; {0}\".format(e))\n except KeyboardInterrupt:\n print(\"Interrupted by User Keyboard\")\n pass\n\n \ndef speech():\n r = sr.Recognizer()\n with sr.Microphone() as source:\n while True:\n # Now we wake the LEDs to indicate the optimum moment now when the user can speak\n pixels.wakeup()\n try:\n r.adjust_for_ambient_noise(source)\n audio_stream = r.listen(source)\n print(\"Waiting for user to speak...\")\n try:\n speech_text = r.recognize_google(audio_stream)\n pixels.off()\n print(\"Google Speech Recognition thinks you said \" + speech_text)\n pixels.think()\n return speech_text\n except sr.UnknownValueError:\n pixels.think()\n print(\"Google Speech Recognition could not understand audio\")\n understand_error = AudioSegment.silent(duration=1000) + AudioSegment.from_mp3(\"sounds/understand.mp3\")\n play(understand_error)\n time.sleep(4)\n except sr.RequestError as e:\n pixels.think()\n print(f\"Could not request results from Google Speech Recognition service; {e}\")\n audio_response = AudioSegment.silent(duration=1000) + AudioSegment.from_mp3(\"sounds/google_issue.mp3\")\n play(audio_response)\n except KeyboardInterrupt:\n print(\"Interrupted by User Keyboard\")\n break # This allows the user to still manually exit the loop with a keyboard interrupt\n\n \ndef chatgpt_response(prompt):\n if prompt is not None:\n # Add a holding messsage like the one below to deal with current TTS delays until such time that TTS can be streamed due to initial buffering how pydub handles audio in memory\n silence = AudioSegment.silent(duration=1000) \n holding_audio_response = silence + AudioSegment.from_mp3(\"sounds/holding.mp3\")\n play(holding_audio_response)\n # send the converted audio text to chatgpt\n response = client.chat.completions.create(\n model=model_engine,\n messages=[{\"role\": \"system\", \"content\": pre_prompt},\n {\"role\": \"user\", \"content\": prompt}],\n max_tokens=400,\n n=1,\n temperature=0.7,\n )\n # Whilst we are waiting for the response, we can play a checking message to improve the user experience.\n checking_on_that = silence + AudioSegment.from_mp3(\"sounds/checking.mp3\")\n play(checking_on_that)\n return response\n else:\n return None\n \ndef generate_audio_file(message):\n speech_file_path = Path(__file__).parent / \"response.mp3\"\n response = client.audio.speech.create(\n model=\"tts-1\",\n voice=\"fable\",\n input=message\n)\n response.stream_to_file(speech_file_path)\n \ndef play_wake_up_audio():\n # play the audio file and wake speaking LEDs\n pixels.speak()\n audio_response = silence + AudioSegment.from_mp3(\"response.mp3\")\n play(audio_response)\n\ndef main():\n # run the program\n # Indicate to the user that the device is ready\n pixels.wakeup()\n device_on = silence + AudioSegment.from_mp3(\"sounds/on.mp3\")\n play(device_on)\n # Play the \"Hello\" audio file to welcome the user\n hello = silence + AudioSegment.from_mp3(\"sounds/hello.mp3\")\n play(hello)\n while True:\n if recognise_speech():\n prompt = speech()\n print(f\"This is the prompt being sent to OpenAI: {prompt}\")\n response = chatgpt_response(prompt)\n if response is not None:\n message = response.choices[0].message.content\n print(message)\n generate_audio_file(message)\n play_wake_up_audio()\n pixels.off()\n else:\n print(\"No prompt to send to OpenAI\")\n # We continue to listen for the wake word\n else:\n print(\"Speech was not recognised\")\n pixels.off()\n \nif __name__ == \"__main__\":\n main()\n"
},
{
"path": "pi.py",
"content": "#!/usr/bin/env python3.9\nimport os\nimport subprocess\nfrom openai import OpenAI\nimport pyaudio\nimport alsaaudio\nfrom datetime import datetime\nimport speech_recognition as sr\nfrom gtts import gTTS\nfrom dotenv import load_dotenv\nimport apa102\nimport threading\nfrom gpiozero import LED\ntry:\n import queue as Queue\nexcept ImportError:\n import Queue as Queue\nfrom alexa_led_pattern import AlexaLedPattern\nfrom pathlib import Path\nfrom pydub import AudioSegment\nfrom pydub.playback import play as pydub_play\nimport time\nimport pvporcupine\nimport struct\nfrom picamera2 import Picamera2\nimport base64\nfrom langchain_community.tools import TavilySearchResults\nfrom langchain.agents import AgentType, initialize_agent\nfrom langchain_openai import ChatOpenAI\nfrom langchain.schema import SystemMessage\n# Set the working directory for Pi if you want to run this code via rc.local script so that it is automatically running on Pi startup. Remove this line if you have installed this project in a different directory.\nos.chdir('/home/pi/ChatGPT-OpenAI-Smart-Speaker')\n\n# We add 0.5 second silence globally due to initial buffering how pydub handles audio in memory\nsilence = AudioSegment.silent(duration=500)\n\n# This is our pre-prompt configuration to precede the user's question to enable OpenAI to understand that it's acting as a smart speaker and add any other required information. We will send this in the OpenAI call as part of the system content in messages.\npre_prompt = \"You are a helpful smart speaker called Jeffers! Please respond with short and concise answers to the following user question and always remind the user at the end to say your name again to continue the conversation:\"\n\n# Load your keys and tokens here\nload_dotenv()\nclient = OpenAI(api_key=os.environ.get(\"OPENAI_API_KEY\"))\ntry:\n TAVILY_API_KEY = os.environ.get(\"TAVILY_API_KEY\")\n print(f\"Tavily search API key found\")\nexcept:\n print(\"Tavily search API key not found.\")\n tavily_key_not_found = silence + AudioSegment.from_mp3(\"sounds/tavily_key_error.mp3\")\n TAVILY_API_KEY = None\n\n# We set the OpenAI model and language settings here for the route that follows general questions and questions with images. This is not for the agent route.\nmodel_engine = \"chatgpt-4o-latest\"\nlanguage = 'en'\n\n# Load the Tavily Search tool which the agent will use to answer questions about weather, news, and recent events.\ntool = TavilySearchResults(\n max_results=20,\n include_answer=True,\n include_raw_content=True,\n include_images=False,\n search_depth=\"advanced\",\n # include_domains = []\n # exclude_domains = []\n)\n\nclass Pixels:\n PIXELS_N = 12\n\n def __init__(self, pattern=AlexaLedPattern):\n self.pattern = pattern(show=self.show)\n self.dev = apa102.APA102(num_led=self.PIXELS_N)\n self.power = LED(5)\n self.power.on()\n self.queue = Queue.Queue()\n self.thread = threading.Thread(target=self._run)\n self.thread.daemon = True\n self.thread.start()\n self.last_direction = None\n\n def wakeup(self, direction=0):\n self.last_direction = direction\n def f():\n self.pattern.wakeup(direction)\n \n self.put(f)\n \n def listen(self):\n if self.last_direction:\n def f():\n self.pattern.wakeup(self.last_direction)\n self.put(f)\n else:\n self.put(self.pattern.listen)\n \n def think(self):\n self.put(self.pattern.think)\n \n def speak(self):\n self.put(self.pattern.speak)\n \n def off(self):\n self.put(self.pattern.off)\n \n def put(self, func):\n self.pattern.stop = True\n self.queue.put(func)\n \n def _run(self):\n while True:\n func = self.queue.get()\n self.pattern.stop = False\n func()\n \n def show(self, data):\n for i in range(self.PIXELS_N):\n self.dev.set_pixel(i, int(data[4*i + 1]), int(data[4*i + 2]), int(data[4*i + 3]))\n \n self.dev.show()\n\n# Instantiate the Pixels class\npixels = Pixels()\n\n# Function to instantiate the PyAudio object for playing audio\ndef play(audio_segment):\n pydub_play(audio_segment)\n\n# This function is called first to detect the wake word \"Jeffers\" and then proceed to listen for the user's question.\ndef detect_wake_word():\n # Here we use the Porcupine wake word detection engine to detect the wake word \"Jeffers\" and then proceed to listen for the user's question.\n porcupine = None\n pa = None\n audio_stream = None\n\n try:\n # Path to the custom wake word .ppn file\n custom_wake_word_path = os.path.join(os.path.dirname(__file__), 'wake_words', 'custom_model/Jeffers_Pi.ppn')\n print(f\"Wake word file path: {custom_wake_word_path}\")\n if not os.path.exists(custom_wake_word_path):\n print(f\"Error: Wake word file not found at {custom_wake_word_path}\")\n \n # Initialize Porcupine with the custom wake word\n # You will need to obtain an access key from Picovoice to use Porcupine (https://console.picovoice.ai/). You can also create your own custom wake word model using the Picovoice Console.\n try:\n porcupine = pvporcupine.create(access_key=os.environ.get(\"ACCESS_KEY\"), keyword_paths=[custom_wake_word_path])\n except pvporcupine.PorcupineInvalidArgumentError as e:\n print(f\"Error creating Porcupine instance: {e}\")\n # Handle the error here\n try:\n pa = pyaudio.PyAudio()\n audio_stream = pa.open(\n rate=porcupine.sample_rate,\n channels=1,\n format=pyaudio.paInt16,\n output_device_index=1,\n input=True,\n input_device_index=pa.get_default_input_device_info()[\"index\"],\n frames_per_buffer=porcupine.frame_length)\n except:\n print(\"Error with audio stream setup.\")\n error_response = silence + AudioSegment.from_mp3(\"sounds/audio_issue.mp3\")\n play(error_response)\n\n while True:\n pcm = audio_stream.read(porcupine.frame_length)\n pcm = struct.unpack_from(\"h\" * porcupine.frame_length, pcm)\n result = porcupine.process(pcm)\n if result >= 0:\n print(\"Wake word detected\")\n return True\n except:\n # Deal with any errors that may occur from using the PicoVoice Service (https://console.picovoice.ai/)\n print(\"Error with wake word detection, Porcupine or the PicoVoice Service.\")\n error_response = silence + AudioSegment.from_mp3(\"sounds/picovoice_issue.mp3\")\n play(error_response)\n\n finally:\n if audio_stream is not None:\n audio_stream.close()\n if pa is not None:\n pa.terminate()\n if porcupine is not None:\n porcupine.delete()\n return False\n\n# This function is called to use the Langchain search agent using the TavilySearchResults tool to answer questions about weather, news, and recent events.\ndef search_agent(speech_text):\n today = datetime.today()\n #! Update this location to your location\n location = \"Colchester, UK\"\n print(f\"Today's date: {today}\")\n print(f\"User's question understood via the search_agent function: {speech_text}\")\n \n search_results = tool.invoke({\n 'query': f\"The current date is {today}, the user is based in {location} and the user wants to know {speech_text}. Keep responses short and concise. Do not respond with links to websites and do not read out website links, search deeper to find the answer. If the question is about weather, please use Celsius as a metric.\"\n })\n \n # Process the search results\n llm = ChatOpenAI(model=\"gpt-4o\", temperature=0.7)\n \n # Prepare the content for the LLM\n content = \"\\n\".join([result['content'] for result in search_results])\n \n # Use the LLM to summarise and extract relevant information\n response = llm.invoke(f\"\"\"\n Based on the following search results, provide a concise and relevant answer to the user's question: \"{speech_text}\"\n \n Search results:\n {content}\n \n Please keep the response short, informative, and directly addressing the user's question. Do not mention sources or include any URLs.\n \"\"\")\n \n return response.content\n\n# This function is called after the wake word is detected to listen for the user's question and then proceed to convert the speech to text.\ndef recognise_speech():\n # Here we use the Google Speech Recognition engine to convert the user's question into text and then send it to OpenAI for a response.\n r = sr.Recognizer()\n with sr.Microphone() as source:\n start_camera = silence + AudioSegment.from_mp3(\"sounds/start_camera.mp3\")\n take_photo = silence + AudioSegment.from_mp3(\"sounds/take_photo.mp3\")\n camera_shutter = silence + AudioSegment.from_mp3(\"sounds/camera_shutter.mp3\")\n agent_search = silence + AudioSegment.from_mp3(\"sounds/agent.mp3\")\n camera_issue = silence + AudioSegment.from_mp3(\"sounds/camera_issue.mp3\")\n print(\"Listening for your question...\")\n audio_stream = r.listen(source, timeout=5, phrase_time_limit=10)\n print(\"Processing your question...\")\n try:\n speech_text = r.recognize_google(audio_stream)\n print(\"Google Speech Recognition thinks you said: \" + speech_text)\n\n # 1. Agent search route\n if any(keyword in speech_text.lower() for keyword in [\"activate search\", \"weather like today\", \"will it rain today\", \"latest news\", \"events are on\"]):\n print(\"Phrase 'activate search', 'weather like today', 'will it rain today', 'latest news', or 'events are on' detected. Using search agent.\")\n play(agent_search)\n agent_response = search_agent(speech_text)\n print(\"Agent response:\", agent_response)\n return agent_response, None, None\n \n # 2. Image capture route\n if \"take a look\" in speech_text.lower() or \"turn on camera\" in speech_text.lower() or \"on the camera\" in speech_text.lower():\n print(\"Phrase 'take a look', 'turn on camera', or 'on the camera' detected.\")\n play(start_camera)\n print(\"Getting ready to capture an image...\")\n play(take_photo)\n try:\n # Updated to use Picamera2, if you want to revert to PiCamera, please follow a previous version of this code and file on our GitHub repository.\n camera = Picamera2()\n # Configure the camera\n camera_config = camera.create_still_configuration(main={\"size\": (640, 480)})\n camera.configure(camera_config)\n camera.start()\n time.sleep(1) # Give the camera time to adjust\n play(camera_shutter)\n image_path = \"captured_image.jpg\"\n camera.capture_file(image_path)\n camera.stop()\n camera.close()\n print(\"Photo captured and saved as captured_image.jpg\")\n return None, image_path, speech_text\n \n except Exception as e:\n print(f\"Pi camera error: {e}\")\n play(camera_issue)\n return None, None, None\n \n # 3. General speech route - no agent or image capture\n return None, None, speech_text\n \n except sr.UnknownValueError:\n print(\"Google Speech Recognition could not understand audio\")\n except sr.RequestError as e:\n print(f\"Could not request results from Google Speech Recognition service; {e}\")\n\n return None, None, None\n\n# This route is called to send the user's general question to OpenAI's ChatGPT model and then play the response to the user.\ndef chatgpt_response(prompt):\n # Here we send the user's question to OpenAI's ChatGPT model and then play the response to the user.\n if prompt is not None:\n try:\n # Add a holding message like the one below to deal with current TTS delays until such time that TTS can be streamed due to initial buffering how pydub handles audio in memory\n silence = AudioSegment.silent(duration=1000)\n holding_audio_response = silence + AudioSegment.from_mp3(\"sounds/holding.mp3\")\n play(holding_audio_response)\n\n # send the converted audio text to chatgpt\n response = client.chat.completions.create(\n model=model_engine,\n messages=[{\"role\": \"system\", \"content\": pre_prompt}, {\"role\": \"user\", \"content\": prompt + \"If the user's question involves browsing the web, local or national current or future events, or event that you are unaware of, news or weather, ALWAYS respond telling them to use the phrase 'activate search' before asking a question. If the users request is to take a photo, ALWAYS respond telling them to use the phrase 'take a look' followed by their request.\"}],\n max_tokens=400,\n n=1,\n temperature=0.7,\n )\n\n # Whilst we are waiting for the response, we can play a checking message to improve the user experience.\n checking_on_that = silence + AudioSegment.from_mp3(\"sounds/checking.mp3\")\n play(checking_on_that)\n\n return response\n except Exception as e:\n # If there is an error, we can play a message to the user to indicate that there was an issue with the API call.\n print(f\"An API error occurred: {str(e)}\")\n error_message = silence + AudioSegment.from_mp3(\"sounds/openai_issue.mp3\")\n play(error_message)\n return None\n else:\n return None\n\n# This route is called to encode the image as base64 when an image is taken.\ndef encode_image(image_path):\n with open(image_path, \"rb\") as image_file:\n return base64.b64encode(image_file.read()).decode('utf-8')\n\n# This route is called if the user's question also includes an image to send to OpenAI's ChatGPT model.\ndef chatgpt_response_with_image(prompt, image_path):\n if prompt is not None:\n try:\n # Add a holding message like the one below to deal with current TTS delays until such time that TTS can be streamed due to initial buffering how pydub handles audio in memory\n silence = AudioSegment.silent(duration=1000)\n holding_audio_response = silence + AudioSegment.from_mp3(\"sounds/holding.mp3\")\n play(holding_audio_response)\n \n # Encode the image as base64\n base64_image = encode_image(image_path)\n \n # Send the converted audio text and image to ChatGPT\n response = client.chat.completions.create(\n model=model_engine,\n messages=[\n {\"role\": \"system\", \"content\": pre_prompt},\n {\n \"role\": \"user\",\n \"content\": [\n {\n \"type\": \"text\",\n \"text\": prompt\n },\n {\n \"type\": \"image_url\",\n \"image_url\": {\n \"url\": f\"data:image/jpeg;base64,{base64_image}\"\n }\n }\n ]\n }\n ],\n max_tokens=400,\n n=1,\n temperature=0.7,\n )\n \n # Whilst we are waiting for the response, we can play a checking message to improve the user experience.\n checking_on_that = silence + AudioSegment.from_mp3(\"sounds/checking.mp3\")\n play(checking_on_that)\n return response\n \n except Exception as e:\n # If there is an error, we can play a message to the user to indicate that there was an issue with the API call.\n print(f\"An API error occurred: {str(e)}\")\n error_message = silence + AudioSegment.from_mp3(\"sounds/openai_issue.mp3\")\n play(error_message)\n return None\n else:\n return None\n\n# This route is called to generate an audio file on demand from the response from OpenAI's ChatGPT model.\ndef generate_audio_file(message):\n # This is a standalone function to generate an audio file from the response from OpenAI's ChatGPT model.\n speech_file_path = Path(__file__).parent / \"response.mp3\"\n response = client.audio.speech.create(\n model=\"tts-1\",\n voice=\"fable\",\n input=message\n)\n response.stream_to_file(speech_file_path)\n \n# This is a standalone function to which we can call to play the audio file and wake speaking LEDs to indicate that the smart speaker is responding to the user.\ndef play_response():\n pixels.speak()\n audio_response = silence + AudioSegment.from_mp3(\"response.mp3\")\n play(audio_response)\n\n# This is the main function that runs the program and controls the flow.\ndef main():\n # This is the main function that runs the program.\n pixels.wakeup()\n device_on = silence + AudioSegment.from_mp3(\"sounds/on.mp3\")\n play(device_on)\n hello = silence + AudioSegment.from_mp3(\"sounds/hello.mp3\")\n play(hello)\n pixels.off()\n while True:\n print(\"Waiting for wake word...\")\n if detect_wake_word():\n pixels.listen() # Indicate that the speaker is listening\n agent_response, image_path, speech_text = recognise_speech()\n if agent_response:\n print(f\"Processed agent response: {agent_response}\") # For debugging\n generate_audio_file(agent_response)\n play_response()\n pixels.off()\n if speech_text:\n if image_path:\n response = chatgpt_response_with_image(speech_text, image_path)\n else:\n response = chatgpt_response(speech_text)\n if response:\n message = response.choices[0].message.content\n print(message)\n generate_audio_file(message)\n play_response()\n pixels.off()\n else:\n print(\"No prompt to send to OpenAI\")\n pixels.off()\n else:\n print(\"Speech was not recognised or there was an error.\")\n pixels.off()\n # After processing (or failure to process), the loop will continue, returning to wake word detection.\nif __name__ == \"__main__\":\n main()"
},
{
"path": "requirements.txt",
"content": "openai\npyaudio\npython-alsa-audio\nSpeechRecognition\ngTTS\npython-dotenv\napa102-pi\ngpiozero\nRPi.GPIO\nalexa-led-pattern\npydub\npvporcupine\npicamera\nlangchain-community\nlangchain\nlangchain-openai\nlangchainhub"
},
{
"path": "requirements_mac.txt",
"content": "# Requirements for simply testing the chat.py script on a Mac\nopenai\npyaudio\nSpeechRecognition\ngTTS\npython-dotenv\npydub\npvporcupine\nlangchain-community\nlangchain\nlangchain-openai\nlangchainhub\nPyObjC\nffmpeg\npydub"
},
{
"path": "test_agent.py",
"content": "from langchain_community.tools.tavily_search import TavilySearchResults\nfrom datetime import datetime\nfrom langchain_openai import ChatOpenAI\nfrom dotenv import load_dotenv\nfrom openai import OpenAI\nimport os\n\nload_dotenv()\n\nmodel = ChatOpenAI(model=\"gpt-4\")\nclient = OpenAI(api_key=os.getenv(\"OPENAI_API_KEY\"))\n\nlocation = \"Colchester, UK\"\ntoday = datetime.today().strftime('%A, %B %d, %Y')\nprint(f\"Today is {today}\")\n\nsearch = TavilySearchResults(max_results=6)\nsearch_results = search.invoke(f\"What local events are not to be missed next week in {location}? The date is {today}.\")\nprint(search_results)\n\n# Now send the results to OpenAI for further processing\nresponse = client.chat.completions.create(\n model=\"gpt-4\",\n messages=[\n {\"role\": \"system\", \"content\": \"Summarise the most up-to-date and applicable information from these search results.\"},\n {\"role\": \"user\", \"content\": str(search_results)} # Convert search_results to a string\n ],\n max_tokens=600,\n n=1,\n temperature=0.7,\n)\nprint(response.choices[0].message.content)"
},
{
"path": "wake_words/custom_model/LICENSE.txt",
"content": "A copy of license terms is available at https://picovoice.ai/docs/terms-of-use/"
}
]