Loading
GroqCall.ai
GroqCall is a proxy server that provides function calls for Groq's lightning-fast Language Processing Unit (LPU) and other AI providers. Additionally, the upcoming FuncyHub will offer a wide range of built-in functions, hosted on the cloud, making it easier to create AI assistants without the need to maintain function schemas in the codebase or execute them through multiple calls.
Check github repo for more info: https://github.com/unclecode/groqcall
Motivation 🚀
Groq is a startup that designs highly specialized processor chips aimed specifically at running inference on large language models. They've introduced what they call the Language Processing Unit (LPU), and the speed is astounding—capable of producing 500 to 800 tokens per second or more. I've become a big fan of Groq and their community;
I admire what they're doing. It feels like after discovering electricity, the next challenge is moving it around quickly and efficiently. Groq is doing just that for Artificial Intelligence, making it easily accessible everywhere. They've opened up their API to the cloud, but as of now, they lack a function call capability.
Unable to wait for this feature, I built a proxy that enables function calls using the OpenAI interface, allowing it to be called from any library. This engineering workaround has proven to be immensely useful in my company for various projects. Here's the link to the GitHub repository where you can explore and play around with it. I've included some examples in this collaboration for you to check out.
Running the Proxy Locally 🖥️
To run this proxy locally on your own machine, follow these steps:
- Clone the GitHub repository:
- Navigate to the project directory:
- Create a virtual environment:
- Activate the virtual environment:
- Install the required libraries:
- Run the FastAPI server:
git clone https://github.com/unclecode/groqcall.gitcd groqcallpython -m venv venvsource venv/bin/activatepip install -r requirements.txt.venv/bin/uvicorn --app-dir app/ main:app --reloadUsing the Pre-built Server 🌐
For your convenience, I have already set up a server that you can use temporarily. This allows you to quickly start using the proxy without having to run it locally.
To use the pre-built server, simply make requests to the following base URL:
https://groqcall.ai/proxy/groq/v1
Exploring GroqCall.ai 🚀
This README is organized into three main sections, each showcasing different aspects of GroqCall.ai:
- Sending POST Requests: Here, I explore the functionality of sending direct POST requests to LLMs using GroqCall.ai. This section highlights the flexibility and control offered by the library when interacting with LLMs.
- FunckyHub: The third section introduces the concept of FunckyHub, a useful feature that simplifies the process of executing functions. With FunckyHub, there is no need to send the function JSON schema explicitly, as the functions are already hosted on the proxy server. This approach streamlines the workflow, allowing developers to obtain results with a single call without having to handle function calls in a production server.
- Using GroqCall with PhiData: In this section, I demonstrate how GroqCall.ai can be seamlessly integrated with other libraries such as my favorite one, the PhiData library, leveraging its built-in tools to connect to LLMs and perform external tool requests.
# The following libraries are optional if you're interested in using PhiData or managing your tools on the client side.
!pip install phidata > /dev/null
!pip install openai > /dev/null
!pip install duckduckgo-search > /dev/null
Sending POST request, with full functions implementation
Check out the example file example_2.py for a full implementation of the following code.
from duckduckgo_search import DDGS
import requests, os
import json
# Here you pass your own GROQ API key
api_key=userdata.get("GROQ_API_KEY")
header = {
"Authorization": f"Bearer {api_key}",
"Content-Type": "application/json"
}
proxy_url = "https://groqcall.ai/proxy/groq/v1/chat/completions"
def duckduckgo_search(query, max_results=None):
"""
Use this function to search DuckDuckGo for a query.
"""
with DDGS() as ddgs:
return [r for r in ddgs.text(query, safesearch='off', max_results=max_results)]
def duckduckgo_news(query, max_results=None):
"""
Use this function to get the latest news from DuckDuckGo.
"""
with DDGS() as ddgs:
return [r for r in ddgs.news(query, safesearch='off', max_results=max_results)]
function_map = {
"duckduckgo_search": duckduckgo_search,
"duckduckgo_news": duckduckgo_news,
}
request = {
"messages": [
{
"role": "system",
"content": "YOU MUST FOLLOW THESE INSTRUCTIONS CAREFULLY.\n<instructions>\n1. Use markdown to format your answers.\n</instructions>"
},
{
"role": "user",
"content": "Whats happening in France? Summarize top stories with sources, very short and concise."
}
],
"model": "mixtral-8x7b-32768",
"tool_choice": "auto",
"tools": [
{
"type": "function",
"function": {
"name": "duckduckgo_search",
"description": "Use this function to search DuckDuckGo for a query.\n\nArgs:\n query(str): The query to search for.\n max_results (optional, default=5): The maximum number of results to return.\n\nReturns:\n The result from DuckDuckGo.",
"parameters": {
"type": "object",
"properties": {
"query": {
"type": "string"
},
"max_results": {
"type": [
"number",
"null"
]
}
}
}
}
},
{
"type": "function",
"function": {
"name": "duckduckgo_news",
"description": "Use this function to get the latest news from DuckDuckGo.\n\nArgs:\n query(str): The query to search for.\n max_results (optional, default=5): The maximum number of results to return.\n\nReturns:\n The latest news from DuckDuckGo.",
"parameters": {
"type": "object",
"properties": {
"query": {
"type": "string"
},
"max_results": {
"type": [
"number",
"null"
]
}
}
}
}
}
]
}
response = requests.post(
proxy_url,
headers=header,
json=request
)
if response.status_code == 200:
res = response.json()
message = res['choices'][0]['message']
tools_response_messages = []
if not message['content'] and 'tool_calls' in message:
for tool_call in message['tool_calls']:
tool_name = tool_call['function']['name']
tool_args = tool_call['function']['arguments']
tool_args = json.loads(tool_args)
if tool_name not in function_map:
print(f"Error: {tool_name} is not a valid function name.")
continue
tool_func = function_map[tool_name]
tool_response = tool_func(**tool_args)
tools_response_messages.append({
"role": "tool", "content": json.dumps(tool_response)
})
if tools_response_messages:
request['messages'] += tools_response_messages
response = requests.post(
proxy_url,
headers=header,
json=request
)
if response.status_code == 200:
res = response.json()
print(res['choices'][0]['message']['content'])
else:
print("Error:", response.status_code, response.text)
else:
print(message['content'])
else:
print("Error:", response.status_code, response.text)
Schema-less Function Call 🤩
Check out the example file example_3.py for a full implementation of the following code.
In this method, we only need to provide the function's name, which consists of two parts, acting as a sort of namespace. The first part identifies the library or toolkit containing the functions, and the second part specifies the function's name, assuming it's already available on the proxy server. I aim to collaborate with the community to incorporate all typical functions, eliminating the need for passing a schema. Without having to handle function calls ourselves, a single request to the proxy enables it to identify and execute the functions, retrieve responses from large language models, and return the results to us. Thanks to Groq, all of this occurs in just seconds.
from duckduckgo_search import DDGS
import requests, os
api_key = userdata.get("GROQ_API_KEY")
header = {
"Authorization": f"Bearer {api_key}",
"Content-Type": "application/json"
}
proxy_url = "https://groqcall.ai/proxy/groq/v1/chat/completions"
request = {
"messages": [
{
"role": "system",
"content": "YOU MUST FOLLOW THESE INSTRUCTIONS CAREFULLY.\n<instructions>\n1. Use markdown to format your answers.\n</instructions>",
},
{
"role": "user",
"content": "Whats happening in France? Summarize top stories with sources, very short and concise. Also please search about the histoy of france as well.",
},
],
"model": "mixtral-8x7b-32768",
"tool_choice": "auto",
"tools": [
{
"type": "function",
"function": {
"name": "duckduck.search",
},
},
{
"type": "function",
"function": {
"name": "duckduck.news",
},
},
],
}
response = requests.post(
proxy_url,
headers=header,
json=request,
)
if response.status_code == 200:
res = response.json()
print(res["choices"][0]["message"]["content"])
else:
print("Error:", response.status_code, response.text)
Using with PhiData
Check out the example file example_1.py for a full implementation of the following code.
PhiData is a favorite of mine for creating AI assistants, thanks to its beautifully simplified interface, unlike the complexity seen in the LangChain library and LlamaIndex. I use it for many projects and want to give kudos to their team. It's open source, and I recommend everyone check it out. You can explore more from this link: https://github.com/phidatahq/phidata.
from google.README import userdata
from phi.llm.openai.like import OpenAILike
from phi.assistant import Assistant
from phi.tools.duckduckgo import DuckDuckGo
import os, json
my_groq = OpenAILike(
model="mixtral-8x7b-32768",
api_key=userdata.get("GROQ_API_KEY"),
base_url="https://groqcall.ai/proxy/groq/v1"
)
assistant = Assistant(
llm=my_groq,
tools=[DuckDuckGo()], show_tool_calls=True, markdown=True
)
assistant.print_response("Whats happening in France? Summarize top stories with sources, very short and concise.", stream=False)
Contributions Welcome! 🙌
I am excited to extend and grow this repository by adding more built-in functions and integrating additional services. If you are interested in contributing to this project and being a part of its development, I would love to collaborate with you! I plan to create a discord channel for this project, where we can discuss ideas, share knowledge, and work together to enhance the repository.
Here's how you can get involved:
- Fork the repository and create your own branch.
- Implement new functions, integrate additional services, or make improvements to the existing codebase.
- Test your changes to ensure they work as expected.
- Submit a pull request describing the changes you have made and why they are valuable.
If you have any ideas, suggestions, or would like to discuss potential contributions, feel free to reach out to me. You can contact me through the following channels:
- Twitter (X): @unclecode
- Email: unclecode@kidocode.com
I'm open to collaboration and excited to see how we can work together to enhance this project and provide value to the community. Let's connect and explore how we can help each other!
Together, let's make this repository even more awesome! 🚀