Repository: joshpxyne/gpt-migrate
Branch: main
Commit: 262cabb5e8da
Files: 61
Total size: 86.6 KB
Directory structure:
gitextract_fjcdwyzl/
├── .gitignore
├── CODE_OF_CONDUCT.md
├── LICENSE
├── README.md
├── TERMS.md
├── benchmarks/
│ ├── flask-cpp/
│ │ └── source/
│ │ ├── .gitignore
│ │ ├── app.py
│ │ ├── db.py
│ │ ├── requirements.txt
│ │ └── storage/
│ │ └── items.json
│ ├── flask-fastapi/
│ │ └── source/
│ │ ├── .gitignore
│ │ ├── app.py
│ │ ├── db.py
│ │ ├── requirements.txt
│ │ └── storage/
│ │ └── items.json
│ ├── flask-nodejs/
│ │ └── source/
│ │ ├── .gitignore
│ │ ├── app.py
│ │ ├── db.py
│ │ ├── requirements.txt
│ │ └── storage/
│ │ └── items.json
│ └── flask-rust/
│ └── source/
│ ├── .gitignore
│ ├── app.py
│ ├── db.py
│ ├── requirements.txt
│ └── storage/
│ └── items.json
├── gpt_migrate/
│ ├── ai.py
│ ├── config.py
│ ├── main.py
│ ├── memory/
│ │ └── external_dependencies
│ ├── parser.py
│ ├── prompts/
│ │ ├── HIERARCHY
│ │ ├── p1_guidelines/
│ │ │ └── guidelines
│ │ ├── p2_actions/
│ │ │ └── write_code
│ │ ├── p3_debug/
│ │ │ ├── create_file
│ │ │ ├── debug_file
│ │ │ ├── debug_target_docker
│ │ │ ├── debug_testfile
│ │ │ ├── human_intervention
│ │ │ ├── identify_action
│ │ │ ├── identify_file
│ │ │ └── move_files
│ │ ├── p3_migrate/
│ │ │ ├── 1_get_external_deps
│ │ │ ├── 2_get_internal_deps
│ │ │ ├── 3_write_migration
│ │ │ ├── 4_add_docker_requirements
│ │ │ ├── 5_refine_target_docker
│ │ │ └── 6_get_function_signatures
│ │ ├── p3_setup/
│ │ │ └── create_target_docker
│ │ ├── p3_test/
│ │ │ └── create_tests
│ │ └── p4_output_formats/
│ │ ├── file_debug
│ │ ├── filenames
│ │ ├── multi_file
│ │ └── single_file
│ ├── requirements.txt
│ ├── steps/
│ │ ├── __init__.py
│ │ ├── debug.py
│ │ ├── migrate.py
│ │ ├── setup.py
│ │ └── test.py
│ └── utils.py
└── pyproject.toml
================================================
FILE CONTENTS
================================================
================================================
FILE: .gitignore
================================================
**/target/
.DS_Store
**/__pycache__/
================================================
FILE: CODE_OF_CONDUCT.md
================================================
# Code of Conduct for GPT-Migrate
## 1. Purpose
The purpose of this Code of Conduct is to provide guidelines for contributors to the GPT-Migrate project on GitHub. We aim to create a positive and inclusive environment where all participants can contribute and collaborate effectively. By participating in this project, you agree to abide by this Code of Conduct.
## 2. Scope
This Code of Conduct applies to all contributors, maintainers, and users of the GPT-Migrate project. It extends to all project spaces, including but not limited to issues, pull requests, code reviews, comments, and other forms of communication within the project.
## 3. Our Standards
We encourage the following behavior:
* Being respectful and considerate to others
* Actively seeking diverse perspectives
* Providing constructive feedback and assistance
* Demonstrating empathy and understanding
We discourage the following behavior:
* Harassment or discrimination of any kind
* Disrespectful, offensive, or inappropriate language or content
* Personal attacks or insults
* Unwarranted criticism or negativity
## 4. Reporting and Enforcement
If you witness or experience any violations of this Code of Conduct, please report them to the project maintainers by email or other appropriate means. The maintainers will investigate and take appropriate action, which may include warnings, temporary or permanent bans, or other measures as necessary.
Maintainers are responsible for ensuring compliance with this Code of Conduct and may take action to address any violations.
## 5. Acknowledgements
This Code of Conduct is adapted from the [Contributor Covenant](https://www.contributor-covenant.org/version/2/0/code_of_conduct.html).
## 6. Contact
If you have any questions or concerns, please contact the project maintainers.
================================================
FILE: LICENSE
================================================
MIT License
Copyright (c) 2023 Josh Payne
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
================================================
FILE: README.md
================================================
<div align="center">
# ◐ GPT-Migrate ◑
**Easily migrate your codebase from one framework or language to another.**
<p>
<a href="https://github.com/0xpayne/gpt-migrate/blob/main/LICENSE"><img alt="Github License" src="https://img.shields.io/badge/License-MIT-green.svg" /></a>
<a href="https://github.com/0xpayne/gpt-migrate"><img alt="GitHub Repo stars" src="https://img.shields.io/github/stars/0xpayne/gpt-migrate?style=social" /></a>
</p>
<br />
</div>
If you've ever faced the pain of migrating a codebase to a new framework or language, this project is for you.
https://user-images.githubusercontent.com/25165841/250232917-bcc99ce8-99b7-4e3d-a653-f89e163ed825.mp4
Migration is a costly, tedious, and non-trivial problem. Do not trust the current version blindly and please use responsibly. Please also be aware that costs can add up quickly as GPT-Migrate is designed to write (and potentially re-write) the entirety of a codebase.
However, with the collective brilliance of the OSS community and the current state of LLMs, it is also a very tractable problem.
## ⚡️ Usage
1. Install Docker and ensure that it's running. It's also recommended that you use at least GPT-4, preferably GPT-4-32k.
## 📦 Installation using Poetry
1. Install Poetry by following the instructions on the [official Poetry website](https://python-poetry.org/docs/#installation).
2. Once Poetry is installed, navigate to the project directory and install the project dependencies using the following command:
```bash
poetry install
```
This will create a virtual environment and install all the necessary dependencies in that environment.
2. Set your [OpenRouter API key](https://openrouter.ai/docs#api-keys) (default) and/or your [OpenAI API key](https://platform.openai.com/account/api-keys) (to use the OpenAI API directly...in this case, set --model to `gpt-4-32k` or your desired model) and install the python requirements:
`export OPENROUTER_API_KEY=<your key>`
`export OPENAI_API_KEY=<your key>`
`pip install -r requirements.txt`
3. Run the main script with the target language you want to migrate to:
`python main.py --targetlang nodejs`
4. (Optional) If you'd like GPT-Migrate to validate the unit tests it creates against your app before it tests the migrated app with them, please have your existing app exposed and use the `--sourceport` flag. For executing this against the benchmark, open a separate terminal, navigate to the `benchmarks/language-pair/source` directory, and run `python app.py` after installing the requirements. It will expose on port 5000. Use this with the `--sourceport` flag.
By default, this script will execute the flask-nodejs benchmark. You can specify the language, source directory, and many other things using the options guide below.
## 💡 Options
You can customize the behavior of GPT-Migrate by passing the following options to the `main.py` script:
- `--model`: The Large Language Model to be used. Default is `"gpt-4-32k"`.
- `--temperature`: Temperature setting for the AI model. Default is `0`.
- `--sourcedir`: Source directory containing the code to be migrated. Default is `"../benchmarks/flask-nodejs/source"`.
- `--sourcelang`: Source language or framework of the code to be migrated. No default value.
- `--sourceentry`: Entrypoint filename relative to the source directory. For instance, this could be an `app.py` or `main.py` file for Python. Default is `"app.py"`.
- `--targetdir`: Directory where the migrated code will live. Default is `"../benchmarks/flask-nodejs/target"`.
- `--targetlang`: Target language or framework for migration. Default is `"nodejs"`.
- `--operating_system`: Operating system for the Dockerfile. Common options are `'linux'` or `'windows'`. Default is `'linux'`.
- `--testfiles`: Comma-separated list of files that have functions to be tested. For instance, this could be an `app.py` or `main.py` file for a Python app where your REST endpoints are. Include the full relative path. Default is `"app.py"`.
- `--sourceport`: (Optional) Port for testing the unit tests file against the original app. No default value. If not included, GPT-Migrate will not attempt to test the unit tests against your original app.
- `--targetport`: Port for testing the unit tests file against the migrated app. Default is `8080`.
- `--guidelines`: Stylistic or small functional guidelines that you'd like to be followed during the migration. For instance, "Use tabs, not spaces". Default is an empty string.
- `--step`: Step to run. Options are `'setup'`, `'migrate'`, `'test'`, `'all'`. Default is `'all'`.
For example, to migrate a Python codebase to Node.js, you might run:
```bash
python main.py --sourcedir /path/to/my-python-app --sourceentry app.py --targetdir /path/to/my-nodejs-app --targetlang nodejs
```
This will take the Python code in `./my-python-app`, migrate it to Node.js, and write the resulting code to `./my-nodejs-app`.
#### GPT-assisted debugging
https://user-images.githubusercontent.com/25165841/250233075-eff1a535-f40e-42e4-914c-042c69ba9195.mp4
## 🤖 How it Works
For migrating a repo from `--sourcelang` to `--targetlang`...
1. GPT-Migrate first creates a Docker environment for `--targetlang`, which is either passed in or assessed automatically by GPT-Migrate.
2. It evaluates your existing code recursively to identify 3rd-party `--sourcelang` dependencies and selects corresponding `--targetlang` dependencies.
3. It recursively rebuilds new `--targetlang` code from your existing code starting from your designated `--sourceentry` file. This step can be started from with the `--step migrate` option.
4. It spins up the Docker environment with the new codebase, exposing it on `--targetport` and iteratively debugging as needed.
5. It develops unit tests using Python's unittest framework, and optionally tests these against your existing app if it's running and exposed on `--sourceport`, iteratively debugging as needed. This step can be started from with the `--step test` option.
6. It tests the new code on `--targetport` against these unit tests.
7. It iteratively debugs the code for for you with context from logs, error messages, relevant files, and directory structure. It does so by choosing one or more actions (move, create, or edit files) then executing them. If it wants to execute any sort of shell script (moving files around), it will first ask for clearance. Finally, if at any point it gets stuck or the user ends the debugging loop, it will output directions for the user to follow to move to the next step of the migration.
8. The new codebase is completed and exists in `--targetdir`.
### 📝 Prompt Design
Subprompts are organized in the following fashion:
- `HIERARCHY`: this defines the notion of preferences. There are 4 levels of preference, and each level prioritized more highly than the previous one.
- `p1`: Preference Level 1. These are the most general prompts, and consist of broad guidelines.
- `p2`: Preference Level 2. These are more specific prompts, and consist of guidelines for certain types of actions (e.g., best practices and philosophies for writing code).
- `p3`: Preference Level 3. These are even more specific prompts, and consist of directions for specific actions (e.g., creating a certain file, debugging, writing tests).
- `p4`: Preference Level 4. These are the most specific prompts, and consist of formatting for output.
Prompts are a combination of subprompts. This concept of tagging and composability can be extended to other properties as well to make prompts even more robust. This is an area we're highly interested in actively exploring.
In this repo, the `prompt_constructor()` function takes in one or more subprompts and yields a string which may be formatted with variables, for example with `GUIDELINES` being a `p1`, `WRITE_CODE` being a `p2` etc:
```python
prompt = prompt_constructor(HIERARCHY, GUIDELINES, WRITE_CODE, DEBUG_TESTFILE, SINGLEFILE).format(targetlang=targetlang,buggyfile=buggyfile)
```
## 📈 Performance
GPT-Migrate is currently in development alpha and is not yet ready for production use. For instance, on the relatively simple benchmarks, it gets through "easy" languages like python or javascript without a hitch ~50% of the time, and cannot get through more complex languages like C++ or Rust without some human assistance.
## ✅ Benchmarks
We're actively looking to build up a robust benchmark repository. If you have a codebase that you'd like to contribute, please open a PR! The current benchmarks were built from scratch: REST API apps which have a few endpoints and dependency files.
## 🧗 Roadmap
Below are improvements on the to-do list. If you'd like to knock any of these or others out, please submit a PR :)
#### High urgency
- Add logic for model input size limiting based on the window size. See issue [#2](https://github.com/0xpayne/gpt-migrate/issues/2).
#### Med urgency
- Add unit tests to the entire project for better reliability and CI/CD
- Add more benchmark examples, especially larger repos
- Add functionality to let the LLM request access to dependency functions in other files as it debugs
- Add support for other LLMs
#### Low urgency
- Enable internet search requests as the model debugs
- Identify and compile language-specific issues + solve for them
## 📣 Call to Action
We're looking for talented contributors. Whether you have a particular passion about a specific language or framework, want to help in creating a more robust test suite, or generally have interesting ideas on how to make this better, we'd love to have you!
## 🛠 Expert-Assisted Migration
Due to the inflow of requests, we've decided to create a standardized process for helping people with their migrations. If you're a company that needs help with a big migration or an expert that is willing to help with them, please visit the following website: [https://gpt-migrate.com/](https://gpt-migrate.com/)
## Join the conversation on [Twitter](https://twitter.com/joshpxyne/status/1675254164165910528)!
================================================
FILE: TERMS.md
================================================
# Terms of Use
By using GPT-Migrate you are aware of and agree to the below Terms of Use.
GPT-Migrate is an experimental application and is provided "as-is" without any warranty, express or implied. By using this software, you agree to assume all risks associated with its use, including but not limited to data loss, system failure, or any other issues that may arise.
The developers and contributors of this project do not accept any responsibility or liability for any losses, damages, or other consequences that may occur as a result of using this software. You are solely responsible for any decisions and actions taken based on the information provided by GPT-Migrate.
Please note that the use of large language models can be expensive. By utilizing this project, you acknowledge that you are responsible for monitoring and managing your own token usage and the associated costs. It is highly recommended to check your usage regularly and set up any necessary limits or alerts to prevent unexpected charges. This is especially true of GPT-Migrate, which will write and potentially re-write the entirety of a codebase.
As an autonomous experiment, GPT-Migrate may generate code or take actions that are not in line with real-world business practices or legal requirements. It is your responsibility to ensure that any actions or decisions made by the generated code comply with all applicable laws, regulations, and ethical standards. The developers and contributors of this project shall not be held responsible for any consequences arising from the use of this software.
By using GPT-Migrate, you agree to indemnify, defend, and hold harmless the developers, contributors, and any affiliated parties from and against any and all claims, damages, losses, liabilities, costs, and expenses (including reasonable attorneys' fees) arising from your use of this software or your violation of these terms.
More information about OpenAI's terms of use, which apply to GPT-Migrate, can be found [here](https://openai.com/policies/terms-of-use).
================================================
FILE: benchmarks/flask-cpp/source/.gitignore
================================================
ignore_me
**/__pycache__/
================================================
FILE: benchmarks/flask-cpp/source/app.py
================================================
from flask import Flask, request, jsonify
from bcrypt import hashpw, gensalt
from db import read_items, write_items
app = Flask(__name__)
@app.route('/', methods=['GET'])
def hello_world():
return "Hello World!"
@app.route('/grocery_items', methods=['GET'])
def get_grocery_items():
try:
grocery_items = read_items()
items = [{"id": item["id"], "name": item["name"], "price": item["price"]} for item in grocery_items]
return jsonify(items)
except Exception as e:
return e, 500
@app.route('/grocery_items', methods=['POST'])
def add_grocery_item():
try:
new_item = request.json
print(new_item["id"],new_item,flush=True)
grocery_items = read_items()
if new_item not in grocery_items:
grocery_items.append(new_item)
write_items(grocery_items)
return "Successfully added item", 201
except Exception as e:
return e, 500
@app.route('/grocery_items/<int:item_id>', methods=['DELETE'])
def delete_grocery_item(item_id):
try:
grocery_items = read_items()
grocery_items = [item for item in grocery_items if item["id"] != item_id]
write_items(grocery_items)
return "Successfully deleted item", 200
except Exception as e:
return e, 500
@app.route('/hashpassword/<string:password>', methods=['GET'])
def hash_password(password):
try:
return hashpw(password.encode('utf-8'), gensalt()).decode('utf-8')
except Exception as e:
return e, 500
if __name__ == '__main__':
app.run(debug=True)
================================================
FILE: benchmarks/flask-cpp/source/db.py
================================================
import json
def read_items():
with open('storage/items.json') as f:
grocery_items = json.load(f)
return grocery_items
def write_items(grocery_items):
with open('storage/items.json', 'w') as f:
json.dump(grocery_items, f)
================================================
FILE: benchmarks/flask-cpp/source/requirements.txt
================================================
flask
bcrypt
================================================
FILE: benchmarks/flask-cpp/source/storage/items.json
================================================
[{"id": 1, "name": "Milk", "price": 2.5}, {"id": 2, "name": "Eggs", "price": 3.0}, {"id": 3, "name": "Bread", "price": 1.5}]
================================================
FILE: benchmarks/flask-fastapi/source/.gitignore
================================================
ignore_me
**/__pycache__/
================================================
FILE: benchmarks/flask-fastapi/source/app.py
================================================
from flask import Flask, request, jsonify
from bcrypt import hashpw, gensalt
from db import read_items, write_items
app = Flask(__name__)
@app.route('/', methods=['GET'])
def hello_world():
return "Hello World!"
@app.route('/grocery_items', methods=['GET'])
def get_grocery_items():
try:
grocery_items = read_items()
items = [{"id": item["id"], "name": item["name"], "price": item["price"]} for item in grocery_items]
return jsonify(items)
except Exception as e:
return e, 500
@app.route('/grocery_items', methods=['POST'])
def add_grocery_item():
try:
new_item = request.json
print(new_item["id"],new_item,flush=True)
grocery_items = read_items()
if new_item not in grocery_items:
grocery_items.append(new_item)
write_items(grocery_items)
return "Successfully added item", 201
except Exception as e:
return e, 500
@app.route('/grocery_items/<int:item_id>', methods=['DELETE'])
def delete_grocery_item(item_id):
try:
grocery_items = read_items()
grocery_items = [item for item in grocery_items if item["id"] != item_id]
write_items(grocery_items)
return "Successfully deleted item", 200
except Exception as e:
return e, 500
@app.route('/hashpassword/<string:password>', methods=['GET'])
def hash_password(password):
try:
return hashpw(password.encode('utf-8'), gensalt()).decode('utf-8')
except Exception as e:
return e, 500
if __name__ == '__main__':
app.run(debug=True)
================================================
FILE: benchmarks/flask-fastapi/source/db.py
================================================
import json
def read_items():
with open('storage/items.json') as f:
grocery_items = json.load(f)
return grocery_items
def write_items(grocery_items):
with open('storage/items.json', 'w') as f:
json.dump(grocery_items, f)
================================================
FILE: benchmarks/flask-fastapi/source/requirements.txt
================================================
flask
bcrypt
================================================
FILE: benchmarks/flask-fastapi/source/storage/items.json
================================================
[{"id": 1, "name": "Milk", "price": 2.5}, {"id": 2, "name": "Eggs", "price": 3.0}, {"id": 3, "name": "Bread", "price": 1.5}]
================================================
FILE: benchmarks/flask-nodejs/source/.gitignore
================================================
ignore_me
**/__pycache__/
================================================
FILE: benchmarks/flask-nodejs/source/app.py
================================================
from flask import Flask, request, jsonify
from bcrypt import hashpw, gensalt
from db import read_items, write_items
app = Flask(__name__)
@app.route('/', methods=['GET'])
def hello_world():
return "Hello World!"
@app.route('/grocery_items', methods=['GET'])
def get_grocery_items():
try:
grocery_items = read_items()
items = [{"id": item["id"], "name": item["name"], "price": item["price"]} for item in grocery_items]
return jsonify(items)
except Exception as e:
return e, 500
@app.route('/grocery_items', methods=['POST'])
def add_grocery_item():
try:
new_item = request.json
print(new_item["id"],new_item,flush=True)
grocery_items = read_items()
if new_item not in grocery_items:
grocery_items.append(new_item)
write_items(grocery_items)
return "Successfully added item", 201
except Exception as e:
return e, 500
@app.route('/grocery_items/<int:item_id>', methods=['DELETE'])
def delete_grocery_item(item_id):
try:
grocery_items = read_items()
grocery_items = [item for item in grocery_items if item["id"] != item_id]
write_items(grocery_items)
return "Successfully deleted item", 200
except Exception as e:
return e, 500
@app.route('/hashpassword/<string:password>', methods=['GET'])
def hash_password(password):
try:
return hashpw(password.encode('utf-8'), gensalt()).decode('utf-8')
except Exception as e:
return e, 500
if __name__ == '__main__':
app.run(debug=True)
================================================
FILE: benchmarks/flask-nodejs/source/db.py
================================================
import json
def read_items():
with open('storage/items.json') as f:
grocery_items = json.load(f)
return grocery_items
def write_items(grocery_items):
with open('storage/items.json', 'w') as f:
json.dump(grocery_items, f)
================================================
FILE: benchmarks/flask-nodejs/source/requirements.txt
================================================
flask
bcrypt
================================================
FILE: benchmarks/flask-nodejs/source/storage/items.json
================================================
[{"id": 1, "name": "Milk", "price": 2.5}, {"id": 2, "name": "Eggs", "price": 3.0}, {"id": 3, "name": "Bread", "price": 1.5}]
================================================
FILE: benchmarks/flask-rust/source/.gitignore
================================================
ignore_me
**/__pycache__/
================================================
FILE: benchmarks/flask-rust/source/app.py
================================================
from flask import Flask, request, jsonify
from bcrypt import hashpw, gensalt
from db import read_items, write_items
app = Flask(__name__)
@app.route('/', methods=['GET'])
def hello_world():
return "Hello World!"
@app.route('/grocery_items', methods=['GET'])
def get_grocery_items():
try:
grocery_items = read_items()
items = [{"id": item["id"], "name": item["name"], "price": item["price"]} for item in grocery_items]
return jsonify(items)
except Exception as e:
return e, 500
@app.route('/grocery_items', methods=['POST'])
def add_grocery_item():
try:
new_item = request.json
print(new_item["id"],new_item,flush=True)
grocery_items = read_items()
if new_item not in grocery_items:
grocery_items.append(new_item)
write_items(grocery_items)
return "Successfully added item", 201
except Exception as e:
return e, 500
@app.route('/grocery_items/<int:item_id>', methods=['DELETE'])
def delete_grocery_item(item_id):
try:
grocery_items = read_items()
grocery_items = [item for item in grocery_items if item["id"] != item_id]
write_items(grocery_items)
return "Successfully deleted item", 200
except Exception as e:
return e, 500
@app.route('/hashpassword/<string:password>', methods=['GET'])
def hash_password(password):
try:
return hashpw(password.encode('utf-8'), gensalt()).decode('utf-8')
except Exception as e:
return e, 500
if __name__ == '__main__':
app.run(debug=True)
================================================
FILE: benchmarks/flask-rust/source/db.py
================================================
import json
def read_items():
with open('storage/items.json') as f:
grocery_items = json.load(f)
return grocery_items
def write_items(grocery_items):
with open('storage/items.json', 'w') as f:
json.dump(grocery_items, f)
================================================
FILE: benchmarks/flask-rust/source/requirements.txt
================================================
flask
bcrypt
================================================
FILE: benchmarks/flask-rust/source/storage/items.json
================================================
[{"id": 1, "name": "Milk", "price": 2.5}, {"id": 2, "name": "Eggs", "price": 3.0}, {"id": 3, "name": "Bread", "price": 1.5}]
================================================
FILE: gpt_migrate/ai.py
================================================
from langchain.chat_models import ChatOpenAI
from config import OPENAI_API_KEY
import os
import openai
from utils import parse_code_string
from litellm import completion
openai.api_key = os.getenv("OPENAI_API_KEY")
class AI:
def __init__(self, model='openrouter/openai/gpt-4-32k', temperature=0.1, max_tokens=10000):
self.temperature = temperature
self.max_tokens = max_tokens
self.model_name = model
try:
_ = ChatOpenAI(model_name=model) # check to see if model is available to user
except Exception as e:
print(e)
self.model_name = "gpt-3.5-turbo"
def write_code(self, prompt):
message=[{"role": "user", "content": str(prompt)}]
response = completion(
messages=message,
stream=False,
model=self.model_name,
max_tokens=self.max_tokens,
temperature=self.temperature
)
if response["choices"][0]["message"]["content"].startswith("INSTRUCTIONS:"):
return ("INSTRUCTIONS:","",response["choices"][0]["message"]["content"][14:])
else:
code_triples = parse_code_string(response["choices"][0]["message"]["content"])
return code_triples
def run(self, prompt):
message=[{"role": "user", "content": str(prompt)}]
response = completion(
messages=message,
stream=True,
model=self.model_name,
max_tokens=self.max_tokens,
temperature=self.temperature
)
chat = ""
for chunk in response:
delta = chunk["choices"][0]["delta"]
msg = delta.get("content", "")
chat += msg
return chat
================================================
FILE: gpt_migrate/config.py
================================================
import os
'''
Environment variables
'''
OPENAI_API_KEY = os.environ.get("OPENAI_API_KEY")
'''
Global variables
'''
MAX_ERROR_MESSAGE_CHARACTERS = 5000
MAX_DOCKER_LOG_CHARACTERS = 2000
'''
Prompt directory
'''
HIERARCHY = "HIERARCHY"
GUIDELINES = "p1_guidelines/guidelines"
WRITE_CODE = "p2_actions/write_code"
CREATE_DOCKER = "p3_setup/create_target_docker"
GET_EXTERNAL_DEPS = "p3_migrate/1_get_external_deps"
GET_INTERNAL_DEPS = "p3_migrate/2_get_internal_deps"
WRITE_MIGRATION = "p3_migrate/3_write_migration"
ADD_DOCKER_REQUIREMENTS = "p3_migrate/4_add_docker_requirements"
REFINE_DOCKERFILE = "p3_migrate/5_refine_target_docker"
GET_FUNCTION_SIGNATURES = "p3_migrate/6_get_function_signatures"
CREATE_TESTS = "p3_test/create_tests"
DEBUG_DOCKERFILE = "p3_debug/debug_target_docker"
IDENTIFY_ACTION = "p3_debug/identify_action"
MOVE_FILES = "p3_debug/move_files"
CREATE_FILE = "p3_debug/create_file"
IDENTIFY_FILE = "p3_debug/identify_file"
DEBUG_FILE = "p3_debug/debug_file"
DEBUG_TESTFILE = "p3_debug/debug_testfile"
HUMAN_INTERVENTION = "p3_debug/human_intervention"
MULTIFILE = "p4_output_formats/multi_file"
SINGLEFILE = "p4_output_formats/single_file"
FILENAMES = "p4_output_formats/filenames"
'''
Living list of types of files that should be excluded from being copied over
'''
EXCLUDED_FILES = [
# Docker
'Dockerfile',
# Python
'requirements.txt',
'__pycache__/',
# JS
'package.json',
'package-lock.json',
'yarn.lock',
'node_modules/'
# Rust
'Cargo.toml'
# TODO: add more
]
'''
Living list of file extensions that should be copied over
'''
INCLUDED_EXTENSIONS = (
'.env',
'.txt',
'.json',
'.csv',
'.rdb',
'.db'
# TODO: add more
)
TREE_SITTER_REPO_STUB = "https://github.com/tree-sitter/tree-sitter-"
EXTENSION_TO_TREE_SITTER_GRAMMAR_REPO = {
'py': TREE_SITTER_REPO_STUB + "python",
'js': TREE_SITTER_REPO_STUB + "javascript",
'java': TREE_SITTER_REPO_STUB + "java",
'rb': TREE_SITTER_REPO_STUB + "ruby",
'php': TREE_SITTER_REPO_STUB + "php",
'cs': TREE_SITTER_REPO_STUB + "c-sharp",
'go': TREE_SITTER_REPO_STUB + "go",
'rs': TREE_SITTER_REPO_STUB + "rust",
'cpp': TREE_SITTER_REPO_STUB + "cpp",
'cc': TREE_SITTER_REPO_STUB + "cpp",
'cxx': TREE_SITTER_REPO_STUB + "cpp",
'c': TREE_SITTER_REPO_STUB + "c",
'swift': TREE_SITTER_REPO_STUB + "swift",
'scala': TREE_SITTER_REPO_STUB + "scala",
'ts': TREE_SITTER_REPO_STUB + "typescript",
'tsx': TREE_SITTER_REPO_STUB + "typescript",
'jsx': TREE_SITTER_REPO_STUB + "javascript",
'hs': TREE_SITTER_REPO_STUB + "haskell",
'jl': TREE_SITTER_REPO_STUB + "julia",
}
EXTENSION_TO_LANGUAGE = {
'py': 'Python',
'js': 'JavaScript',
'java': 'Java',
'rb': 'Ruby',
'php': 'PHP',
'cs': 'C#',
'go': 'Go',
'rs': 'Rust',
'cpp': 'C++',
'cc': 'C++',
'cxx': 'C++',
'c': 'C',
'swift': 'Swift',
'm': 'Objective-C',
'kt': 'Kotlin',
'scala': 'Scala',
'pl': 'Perl',
'pm': 'Perl',
'r': 'R',
'lua': 'Lua',
'groovy': 'Groovy',
'ts': 'TypeScript',
'tsx': 'TypeScript',
'jsx': 'JavaScript',
'dart': 'Dart',
'elm': 'Elm',
'erl': 'Erlang',
'ex': 'Elixir',
'fs': 'F#',
'hs': 'Haskell',
'jl': 'Julia',
'nim': 'Nim',
}
================================================
FILE: gpt_migrate/main.py
================================================
import typer
from utils import detect_language, build_directory_structure
import os
import time as time
from collections import defaultdict
from ai import AI
from steps.setup import create_environment
from steps.migrate import get_dependencies, write_migration, add_env_files
from steps.test import run_dockerfile, create_tests, validate_tests, run_test
from steps.debug import debug_testfile, debug_error
app = typer.Typer()
class Globals:
def __init__(self, sourcedir, targetdir, sourcelang, targetlang, sourceentry, source_directory_structure, operating_system, testfiles, sourceport, targetport, guidelines, ai):
self.sourcedir = sourcedir
self.targetdir = targetdir
self.sourcelang = sourcelang
self.targetlang = targetlang
self.sourceentry = sourceentry
self.source_directory_structure = source_directory_structure
self.operating_system=operating_system
self.testfiles = testfiles
self.sourceport = sourceport
self.targetport = targetport
self.guidelines = guidelines
self.ai = ai
@app.command()
def main(
model: str = typer.Option('openrouter/openai/gpt-4-32k', help="Large Language Model to be used. Default is 'openrouter/openai/gpt-4-32k'. To use OpenAI directly with your API key, use 'gpt-4-32k'."),
temperature: float = typer.Option(0, help="Temperature setting for the AI model."),
sourcedir: str = typer.Option("../benchmarks/flask-nodejs/source", help="Source directory containing the code to be migrated."),
sourcelang: str = typer.Option(None, help="Source language or framework of the code to be migrated."),
sourceentry: str = typer.Option("app.py", help="Entrypoint filename relative to the source directory. For instance, this could be an app.py or main.py file for Python."),
targetdir: str = typer.Option("../benchmarks/flask-nodejs/target", help="Directory where the migrated code will live."),
targetlang: str = typer.Option("nodejs", help="Target language or framework for migration."),
operating_system: str = typer.Option("linux", help="Operating system for the Dockerfile. Common options are 'linux' or 'windows'."),
testfiles: str = typer.Option("app.py", help="Comma-separated list of files that have functions to be tested. For instance, this could be an app.py or main.py file for Python app where your REST endpoints are. Include the full relative path."),
sourceport: int = typer.Option(None, help="(Optional) port for testing the unit tests file against the original app."),
targetport: int = typer.Option(8080, help="Port for testing the unit tests file against the migrated app."),
guidelines: str = typer.Option("", help="Stylistic or small functional guidelines that you'd like to be followed during the migration. For instance, \"Use tabs, not spaces\"."),
step: str = typer.Option("all", help="Step to run. Options are 'setup', 'migrate', 'test', 'all'.")
):
ai = AI(
model=model,
temperature=temperature,
)
sourcedir = os.path.abspath(sourcedir)
targetdir = os.path.abspath(targetdir)
os.makedirs(targetdir, exist_ok=True)
detected_language = detect_language(sourcedir) if not sourcelang else sourcelang
if not sourcelang:
if detected_language:
is_correct = typer.confirm(f"Is your source project a {detected_language} project?")
if is_correct:
sourcelang = detected_language
else:
sourcelang = typer.prompt("Please enter the correct language for the source project")
else:
sourcelang = typer.prompt("Unable to detect the language of the source project. Please enter it manually")
if not os.path.exists(os.path.join(sourcedir, sourceentry)):
sourceentry = typer.prompt("Unable to find the entrypoint file. Please enter it manually. This must be a file relative to the source directory.")
source_directory_structure = build_directory_structure(sourcedir)
globals = Globals(sourcedir, targetdir, sourcelang, targetlang, sourceentry, source_directory_structure, operating_system, testfiles, sourceport, targetport, guidelines, ai)
typer.echo(typer.style(f"◐ Reading {sourcelang} project from directory '{sourcedir}', with entrypoint '{sourceentry}'.", fg=typer.colors.BLUE))
time.sleep(0.3)
typer.echo(typer.style(f"◑ Outputting {targetlang} project to directory '{targetdir}'.", fg=typer.colors.BLUE))
time.sleep(0.3)
typer.echo(typer.style("Source directory structure: \n\n" + source_directory_structure, fg=typer.colors.BLUE))
''' 1. Setup '''
if step in ['setup', 'all']:
# Set up environment (Docker)
create_environment(globals)
''' 2. Migration '''
if step in ['migrate', 'all']:
target_deps_per_file = defaultdict(list)
def migrate(sourcefile, globals, parent_file=None):
# recursively work through each of the files in the source directory, starting with the entrypoint.
internal_deps_list, external_deps_list = get_dependencies(sourcefile=sourcefile,globals=globals)
for dependency in internal_deps_list:
migrate(dependency, globals, parent_file=sourcefile)
file_name = write_migration(sourcefile, external_deps_list, target_deps_per_file.get(sourcefile), globals)
target_deps_per_file[parent_file].append(file_name)
migrate(sourceentry, globals)
add_env_files(globals)
''' 3. Testing '''
if step in ['test', 'all']:
while True:
result = run_dockerfile(globals)
if result=="success": break
debug_error(result,"",globals)
for testfile in globals.testfiles.split(','):
generated_testfile = create_tests(testfile,globals)
if globals.sourceport:
while True:
result = validate_tests(generated_testfile, globals)
time.sleep(0.3)
if result=="success": break
debug_testfile(result,testfile,globals)
while True:
result = run_test(generated_testfile, globals)
if result=="success": break
debug_error(result,globals.testfiles,globals)
run_dockerfile(globals)
time.sleep(1) # wait for docker to spin up
typer.echo(typer.style("All tests complete. Ready to rumble. 💪", fg=typer.colors.GREEN))
if __name__ == "__main__":
app()
================================================
FILE: gpt_migrate/memory/external_dependencies
================================================
rocket
serde
serde_json
bcrypt
rusqlite
serde_json
================================================
FILE: gpt_migrate/parser.py
================================================
import subprocess
import typer
from yaspin import yaspin
from pathlib import Path
from tree_sitter import Language, Parser, Node
from collections.abc import Iterator
from config import EXTENSION_TO_TREE_SITTER_GRAMMAR_REPO, EXTENSION_TO_LANGUAGE
def decompose_file(file_path: str) -> Iterator[Node]:
# Do a first-level parse tree decomposition of the file at file_path
with yaspin(text="Decomposing file", spinner="dots") as spinner:
repo_url = EXTENSION_TO_TREE_SITTER_GRAMMAR_REPO.get(file_path.split('.')[-1])
if not repo_url:
success_text = typer.style("Couldn't find tree-sitter grammar for programming language {}. Aborting decomposition of file.".format(EXTENSION_TO_LANGUAGE.get(file_path.split('.')[-1])), fg=typer.colors.RED)
typer.echo(success_text)
repo_name = repo_url.split('/')[-1]
if not Path("cache/tree-sitter/" + repo_name).exists():
Path("cache/tree-sitter").mkdir(parents=True, exist_ok=True)
result = subprocess.run(["git", "clone", repo_url], cwd="cache/tree-sitter", stdout=subprocess.PIPE, stderr=subprocess.STDOUT, check=True, text=True)
grammar_lib_path = Path("cache/tree-sitter/my-languages.so")
if grammar_lib_path.exists():
grammar_lib_path.unlink()
Language.build_library(
'cache/tree-sitter/my-languages.so',
['cache/tree-sitter/' + repo_name]
)
lang = Language('cache/tree-sitter/my-languages.so', repo_url.split('-')[-1])
parser = Parser()
parser.set_language(lang)
with open(file_path) as f:
source_code = f.read()
tree = parser.parse(bytes(source_code, "utf8"))
root_node = tree.root_node
for child in root_node.children:
yield child
spinner.ok("✅ ")
================================================
FILE: gpt_migrate/prompts/HIERARCHY
================================================
The following prompt is a composition of prompt sections, each with different preference levels. Higher preference levels override lower preference levels. The lowest preference level, PREFERENCE LEVEL 1, will likely be a broad guideline of some sort. Prompt sections with higher preference levels are likely to be more specific instructions that would override sections with lower preference levels. Each prompt section will start with "PREFERENCE LEVEL (level)".
================================================
FILE: gpt_migrate/prompts/p1_guidelines/guidelines
================================================
\n\n PREFERENCE LEVEL 1
Here are the guidelines for this prompt:
1. Follow the output instructions precisely and do not make any assumptions. Your output will not be read by a human; it will be directly input into a computer for literal processing. Adding anything else or deviating from the instructions will cause the output to fail.
2. Think through the answer to each prompt step by step to ensure that the output is perfect; there is no room for error.
3. Do not use any libraries, frameworks, or projects that are not well-known and well-documented, unless they are explicitly mentioned in the instructions or in the prompt.
4. In general, use comments in code only sparingly.
================================================
FILE: gpt_migrate/prompts/p2_actions/write_code
================================================
\n\n PREFERENCE LEVEL 2
You are a pragmatic principal engineer at Google. You are about to get instructions for code to write. This code must be as simple and easy to understand, while still fully expressing the functionality required. Please note that the code should be complete and fully functional. No placeholders. However, only write what you are asked to write. For instance, if you're asked to write a function, only write the function; DO NOT include import statements. We will do those separately.
Please strictly follow this styling guideline with no deviations. Variables will always be snake_case; either capital or lowercase. Functions will always be camelCase. Classes will always be PascalCase. Please follow this guideline even if the source code does not follow it.
Finally, please follow these guidelines: {guidelines}
================================================
FILE: gpt_migrate/prompts/p3_debug/create_file
================================================
\n\n PREFERENCE LEVEL 3
You are a principal software engineer at Google. The current app is having trouble running. Consider the below error and current directory structure. It has been determined that creating a new file in the directory may help. Please provide a file with the full relative path name in the format you're about to see.
Finally, the script should only be one line. For multiple commands, please use the && operator.
```
Error message:
```
{error_message}
```
Current directory structure:
```
{target_directory_structure}
```
================================================
FILE: gpt_migrate/prompts/p3_debug/debug_file
================================================
\n\n PREFERENCE LEVEL 3
You are a principal software engineer at Google. We are doing a migration to {targetlang} from {sourcelang}. The current Docker app is having trouble running. Consider the below error and the current version of {file_name}, which is at least partially responsible for this bug. Please rewrite this file to fix the bug.
Error message:
```
{error_message}
```
Docker logs:
```
{docker_logs}
```
Current {file_name}:
```
{old_file_content}
```
Other files that may be relevant:
{relevant_files}
================================================
FILE: gpt_migrate/prompts/p3_debug/debug_target_docker
================================================
\n\n PREFERENCE LEVEL 3
You are a principal software engineer at Google with particular expertise in Docker environments. The current Docker environment is having trouble running. Consider the below error, current Dockerfile, and directory structure. Your job is to create a comprehensive Dockerfile which will allow the app to run in a Docker environment.
Error message:
```
{error_message}
```
Target directory structure:
```
{target_directory_structure}
```
Current Dockerfile:
```
{dockerfile_content}
```
================================================
FILE: gpt_migrate/prompts/p3_debug/debug_testfile
================================================
\n\n PREFERENCE LEVEL 3
You are a principal software engineer at Google. We are writing {file_name} using unittest against the following source file. The test script is reporting the following error(s), see below. Consider the below error and the current version of {file_name}. Please rewrite this file to fix the bug. Try to ensure that for whatever tests you write, the databases involved end up at their original state after running all of the tests. Create one test function per endpoint or testable function.
Error message:
```
{error_message}
```
Current {file_name}:
```
{old_file_content}
```
Source test file(s):
{relevant_files}
================================================
FILE: gpt_migrate/prompts/p3_debug/human_intervention
================================================
\n\n PREFERENCE LEVEL 3
You are a principal software engineer at Google who is pair programming with a partner. You've been given the following error message:
```
{error_message}
```
Here are some relevant files:
{relevant_files}
And finally, here is the current directory structure:
```
{target_directory_structure}
```
Please respond in simple English with a description of the error and a suggested fix.
================================================
FILE: gpt_migrate/prompts/p3_debug/identify_action
================================================
\n\n PREFERENCE LEVEL 3
You are a principal software engineer at Google pair programming with a partner. The current Docker app is having trouble running. Consider the below error and the directory structure. Your job is to identify which action to take from the following: MOVE_FILES, CREATE_FILES, EDIT_FILES. Your output must be a comma-separated list of one or more of the options, for example: MOVE_FILES,EDIT_FILES.
If copying, moving, or deleting a file or directory will help, output "MOVE_FILES".
If creating one or more new files will help you resolve the issue, output "CREATE_FILES".
If editing one or more existing files can help resolve the issue, output "EDIT_FILES".
It's highly important that your output is a comma-separated list of MOVE_FILES, CREATE_FILES, and/or EDIT_FILES. Nothing else.
Error message:
```
{error_message}
```
Directory structure:
```
{target_directory_structure}
```
================================================
FILE: gpt_migrate/prompts/p3_debug/identify_file
================================================
\n\n PREFERENCE LEVEL 3
You are a principal software engineer at Google. The current Docker app is having trouble running. Consider the below error and the directory structure. Your job is to identify which file is responsible for this error. If multiple files, please list them in comma-separated values and include the whole relative path, for example: file1.ext,directory1/file2.ext,file3.ext...
Do not select any files in the gpt_migrate directory, unless the gpt_migrate directory is specifically mentioned below in the error message (in this case, include gpt_migrate/filename). If the error is from one of these, you'll need to fix the correspondingly named file in the main directory.
Error message:
```
{error_message}
```
Docker logs:
```
{docker_logs}
```
Directory structure:
```
{target_directory_structure}
```
The output format should be either a single filename or a comma-separated list of filenames, and nothing else. If there are none that are obvious, please write only "NONE FOUND" and nothing else.
================================================
FILE: gpt_migrate/prompts/p3_debug/move_files
================================================
\n\n PREFERENCE LEVEL 3
You are a principal software engineer at Google. The current app is having trouble running. Consider the below error and current directory structure. It has been determined that moving (or renaming), copying, or removing one or more files from the directory may help. Please provide a complete shell script that will do so using only the commands `cp`, `mv`, or `rm`. We are not able to change directories so please use the full path in all commands and do not use `cd`. This should be written for the {operating_system} operating system. This will be run directly in the terminal.
Finally, the script should only be one line. For multiple commands, please use the && operator.
```
Error message:
```
{error_message}
```
Current directory structure:
```
{target_directory_structure}
```
Full path:
```
{current_full_path}
```
================================================
FILE: gpt_migrate/prompts/p3_migrate/1_get_external_deps
================================================
\n\n PREFERENCE LEVEL 3
You are a principal software engineer at Google with particular expertise migrating codebases from {sourcelang} to {targetlang}. We are doing a migration from {sourcelang} to {targetlang}. Please respond only with a comma-separated list of {targetlang} libraries you would want to have installed in a {targetlang} project based on the following {sourcelang} file:
```
{sourcefile_content}
```
If there are no outside libraries, answer only NONE. If there are libraries, please list them in the following format:
dep1,dep2,dep3...
Please do not include any other information in your answer. The content of your output will be directly read into a file and any deviation will cause this process to fail.
================================================
FILE: gpt_migrate/prompts/p3_migrate/2_get_internal_deps
================================================
\n\n PREFERENCE LEVEL 3
You are a principal software engineer at Google with particular expertise migrating codebases from {sourcelang} to {targetlang}. We are doing a migration from {sourcelang} to {targetlang}. Please respond only with a comma-separated list of any internal dependency files. For instance, if we were in python, `from db import mongo` would probably indicate there is an internal dependency file called db.py. Some may be external; ignore those. For instance, if we were in python, `import pandas` would be an external dependency and it should be ignored. We are currently in {sourcefile}.
Directory structure:
```
{source_directory_structure}
```
{sourcefile}:
```
{sourcefile_content}
```
If there are no internal dependency files, answer only NONE. If there are internal dependency files, please list their paths relative to the root of this directory in the following comma-separated format as such:
dep1.ext,folder1/dep2.ext,dep3.ext...
Please do not include any other information in your answer. The content of your output will be directly read into a file and any deviation will cause this process to fail.
================================================
FILE: gpt_migrate/prompts/p3_migrate/3_write_migration
================================================
\n\n PREFERENCE LEVEL 3
You are a principal software engineer at Google with particular expertise migrating codebases from {sourcelang} to {targetlang}. We are doing a migration from {sourcelang} to {targetlang}. You are allowed to use the following external libraries, but no other external libraries: {external_deps}. You will be given the current target directory structure of the {targetlang} project, the source directory structure of the existing {sourcelang} project, and the contents of the {sourcelang} file. Please use the below code format and name the file, variables, functions, etc. to be consistent with the existing {sourcelang} file where possible. The only exception is if this is an entrypoint file and {targetlang} requires a certain naming convention, such as main.ext etc. For the filename, include the full relative path if applicable. If the {sourcelang} code imports internal libraries from a given location, take special care to preserve this topology in the code you write for the {targetlang} project and use the functions in the internal libraries accordingly. Any port listening should be on 8080. Please ensure that all functions and variables are available to other files that may call them.
Current target directory structure, which is under active development and may have files added later which you can import from:
```
{target_directory_structure}
```
Existing source directory structure:
```
{source_directory_structure}
```
Existing {sourcelang} file, {sourcefile}:
```
{sourcefile_content}
```
Available internal functions in {targetlang}, their signatures and descriptions:
```
{targetlang_function_signatures}
```
================================================
FILE: gpt_migrate/prompts/p3_migrate/4_add_docker_requirements
================================================
\n\n PREFERENCE LEVEL 3
You are a principal software engineer at Google with particular expertise in Docker environments. Consider the below Dockerfile and create a file for external dependencies. Some languages require a specific directory and/or file name, and to address this in the file for external dependencies. The current directory structure is as follows:
```
{target_directory_structure}
```
If this will be an issue for {targetlang}, please handle for this in the external dependencies file.
Dockerfile:
```
{dockerfile_content}
```
External dependencies:
```
{external_deps}
```
================================================
FILE: gpt_migrate/prompts/p3_migrate/5_refine_target_docker
================================================
\n\n PREFERENCE LEVEL 3
You are a principal software engineer at Google with particular expertise in Docker environments. Consider the below Dockerfile, dependencies file, and directory structure. Your job is to create a comprehensive Dockerfile which will allow the app to run in a Docker environment the first time.
Target directory structure:
```
{target_directory_structure}
```
Current Dockerfile:
```
{dockerfile_content}
```
External dependencies, {external_deps_name}:
```
{external_deps_content}
```
================================================
FILE: gpt_migrate/prompts/p3_migrate/6_get_function_signatures
================================================
\n\n PREFERENCE LEVEL 3
You are a principal software engineer at Google with particular expertise migrating codebases from {sourcelang} to {targetlang}. We are doing a migration from {sourcelang} to {targetlang}. As an intermediate step, we have to extract all function signatures from a {targetlang} file. Make sure to include types as well as default arguments. Futhermore, we have to give a concise description of the function. It should be clear how to call the function from the description and signature only. Here is the {targetlang} file:
```
{targetfile_content}
```
If there are no functions, answer only NONE. If there are functions, please respond in JSON format. Here is an example for the structure of the response for a hypothetical Python file.
[
{{
"signature": "is_prime(number: int) -> bool",
"description": "Determines if the number is a prime number",
}},
{{
"signature": "get_current_weather(location: string, unit: string = "celsius") -> int",
"description": "Get the current weather in a given location. The unit can be optionally specified and should be either celsius or fahrenheit.",
}}
]
Please do not include any other information in your answer. The content of your output will be directly read into a file and any deviation will cause this process to fail.
================================================
FILE: gpt_migrate/prompts/p3_setup/create_target_docker
================================================
\n\n PREFERENCE LEVEL 3
Please create a Dockerfile for a generic app in the following framework: {targetlang}. This new app is being transpiled to {targetlang} from {sourcelang}, where the entrypoint file name is {sourceentry}. Please use the same file name besides the extension if in a different language, unless {targetlang} requires a certain naming convention for the entrypoint file, such as main.ext etc. Be sure to include a dependencies installation step with your choice of file name. No need to write any comments. Exposed port should be 8080.
================================================
FILE: gpt_migrate/prompts/p3_test/create_tests
================================================
\n\n PREFERENCE LEVEL 3
You are a principal software engineer at Google. You're responsible for creating a set of tests for this piece of code using Python. The tests should cover each main function - execute an input, check the output, repeat for each function. Assume this is exposed on port {targetport} if applicable. Please write a set of tests that can be executed either as a python file or as a shell script that will validate or invalidate this file. Please ensure that the error messages for the test file are clear and descriptive. Use unittest. Try to ensure that for whatever tests you write, the databases involved end up at their original state after running all of the tests. Create one test function per endpoint or testable function. Finally, in the logs, if the test fails, please have it log or print the response (instead of just asserting something). This will help us debug the issue.
```
{old_file_content}
```
================================================
FILE: gpt_migrate/prompts/p4_output_formats/file_debug
================================================
\n\n PREFERENCE LEVEL 4
We will be using the output you provide as-is to create new files OR provide natural language instructions for how to debug the problem.
If your output is a new file, please be precise and do not include any other text. Your output needs to be ONE file; if your output contains multiple files, it will break the system. Your output should consist ONLY of the file name, language, and code, in the following format:
file_name.ext
```language
CODE
```
If you decide that the problem cannot be fixed by simply changing a file, please provide natural language instructions for how to debug the problem. The format should be "INSTRUCTIONS: " followed by the text. For example:
INSTRUCTIONS: You need to change the file name from "file_name.ext" to "new_file_name.ext" and then run the following command: "python3 file_name.ext".
================================================
FILE: gpt_migrate/prompts/p4_output_formats/filenames
================================================
\n\n PREFERENCE LEVEL 4
Please respond only with a comma-separated list of items:
file.ext,dir1/file2.ext,file3.ext...
Please do not include any other information in your answer. The content of your output will be directly read into a file and any deviation will cause this process to fail. If your list is empty, please instead write only NONE.
================================================
FILE: gpt_migrate/prompts/p4_output_formats/multi_file
================================================
\n\n PREFERENCE LEVEL 4
We will be using the output you provide as-is to create new files, so please be precise and do not include any other text. Your output should consist ONLY of the filename, language, and code, in the following format, separated by three dashes (---) for multiple files as shown:
Filename.ext
```language
CODE
```
---
Filename.ext
```language
CODE
```
---
Filename.ext
```language
CODE
```
---
...
================================================
FILE: gpt_migrate/prompts/p4_output_formats/single_file
================================================
\n\n PREFERENCE LEVEL 4
We will be using the output you provide as-is to create new files, so please be precise and do not include any other text. Your output needs to be ONE file; if your output contains multiple files, it will break the system. Your output should consist ONLY of the file name, language, and code, in the following format:
file_name.ext
```language
CODE
```
================================================
FILE: gpt_migrate/requirements.txt
================================================
typer
langchain
yaspin
openai
tree_sitter
litellm==0.1.213
pydantic==1.10.8
================================================
FILE: gpt_migrate/steps/__init__.py
================================================
================================================
FILE: gpt_migrate/steps/debug.py
================================================
from utils import prompt_constructor, llm_write_file, llm_run, build_directory_structure, construct_relevant_files
from config import HIERARCHY, GUIDELINES, WRITE_CODE, IDENTIFY_ACTION, MOVE_FILES, CREATE_FILE, IDENTIFY_FILE, DEBUG_FILE, DEBUG_TESTFILE, HUMAN_INTERVENTION, SINGLEFILE, FILENAMES, MAX_ERROR_MESSAGE_CHARACTERS, MAX_DOCKER_LOG_CHARACTERS
import os
import typer
import subprocess
def debug_error(error_message,relevant_files,globals):
identify_action_template = prompt_constructor(HIERARCHY, GUIDELINES, IDENTIFY_ACTION)
prompt = identify_action_template.format(error_message=error_message[-min(MAX_ERROR_MESSAGE_CHARACTERS, len(error_message)):],
target_directory_structure=build_directory_structure(globals.targetdir))
actions = llm_run(prompt,
waiting_message=f"Planning actions for debugging...",
success_message="",
globals=globals)
action_list = actions.split(',')
if "MOVE_FILES" in action_list:
if not os.path.exists(os.path.join(globals.targetdir, 'gpt_migrate')):
os.makedirs(os.path.join(globals.targetdir, 'gpt_migrate'))
move_files_template = prompt_constructor(HIERARCHY, GUIDELINES, WRITE_CODE, MOVE_FILES, SINGLEFILE)
prompt = move_files_template.format(error_message=error_message[-min(MAX_ERROR_MESSAGE_CHARACTERS, len(error_message)):],
target_directory_structure=build_directory_structure(globals.targetdir),
current_full_path=globals.targetdir,
operating_system=globals.operating_system,
guidelines=globals.guidelines)
file_name, language, shell_script_content = llm_write_file(prompt,
target_path="gpt_migrate/debug.sh",
waiting_message=f"Writing shell script...",
success_message="Wrote debug.sh based on error message.",
globals=globals)
# execute shell script from file_content using subprocess. Check with user using shell commands before executing.
if typer.confirm("GPT-Migrate wants to run this shell script to debug your application, which is also stored in gpt_migrate/debug.sh: \n\n"+shell_script_content+"\n\nWould you like to run it?"):
try:
result = subprocess.run(["bash", "gpt_migrate/debug.sh"], stdout=subprocess.PIPE, stderr=subprocess.STDOUT, check=True, text=True)
print(result.stdout)
except subprocess.CalledProcessError as e:
print("ERROR: ",e.output)
error_message = e.output
error_text = typer.style("Something isn't right with your shell script. Please ensure it is valid and try again.", fg=typer.colors.RED)
typer.echo(error_text)
raise typer.Exit()
if "EDIT_FILES" in action_list:
if relevant_files != "":
fileslist = globals.testfiles.split(',')
files_to_construct = []
for file_name in fileslist:
with open(os.path.join(globals.sourcedir, file_name), 'r') as file:
file_content = file.read()
files_to_construct.append(("migration_source/"+file_name, file_content))
relevant_files = construct_relevant_files(files_to_construct)
identify_file_template = prompt_constructor(HIERARCHY, GUIDELINES, IDENTIFY_FILE, FILENAMES)
docker_logs = subprocess.run(["docker", "logs", "gpt-migrate"], stdout=subprocess.PIPE, stderr=subprocess.STDOUT, check=True, text=True).stdout
prompt = identify_file_template.format(error_message=error_message[-min(MAX_ERROR_MESSAGE_CHARACTERS, len(error_message)):],
target_directory_structure=build_directory_structure(globals.targetdir),
docker_logs=docker_logs[-min(MAX_DOCKER_LOG_CHARACTERS, len(docker_logs)):]),
file_names = llm_run(prompt,
waiting_message=f"Identifying files to debug...",
success_message="",
globals=globals)
file_name_list = file_names.split(',')
for file_name in file_name_list:
old_file_content = ""
try:
with open(os.path.join(globals.targetdir, file_name), 'r') as file:
old_file_content = file.read()
except:
print("File not found: "+file_name+". Please ensure the file exists and try again. You can resume the debugging process with the `--step test` flag.")
raise typer.Exit()
debug_file_template = prompt_constructor(HIERARCHY, GUIDELINES, WRITE_CODE, DEBUG_FILE, SINGLEFILE)
prompt = debug_file_template.format(error_message=error_message[-min(MAX_ERROR_MESSAGE_CHARACTERS, len(error_message)):],
file_name=file_name,
old_file_content=old_file_content,
targetlang=globals.targetlang,
sourcelang=globals.sourcelang,
docker_logs=docker_logs[-min(MAX_DOCKER_LOG_CHARACTERS, len(docker_logs)):],
relevant_files=relevant_files,
guidelines=globals.guidelines),
_, language, file_content = llm_write_file(prompt,
target_path=file_name,
waiting_message=f"Debugging {file_name}...",
success_message=f"Re-wrote {file_name} based on error message.",
globals=globals)
new_file_content = ""
with open(os.path.join(globals.targetdir, file_name), 'r') as file:
new_file_content = file.read()
if new_file_content==old_file_content:
require_human_intervention(error_message,construct_relevant_files([(file_name, new_file_content)]),globals)
if "CREATE_FILE" in action_list:
create_file_template = prompt_constructor(HIERARCHY, GUIDELINES, WRITE_CODE, CREATE_FILE, SINGLEFILE)
prompt = create_file_template.format(error_message=error_message[-min(MAX_ERROR_MESSAGE_CHARACTERS, len(error_message)):],
target_directory_structure=build_directory_structure(globals.targetdir),
guidelines=globals.guidelines)
new_file_name, language, file_content = llm_write_file(prompt,
waiting_message=f"Creating a new file...",
success_message="",
globals=globals)
success_text = typer.style(f"Created new file {new_file_name}.", fg=typer.colors.GREEN)
typer.echo(success_text)
def debug_testfile(error_message,testfile,globals):
source_file_content = ""
with open(os.path.join(globals.sourcedir, testfile), 'r') as file:
source_file_content = file.read()
relevant_files = construct_relevant_files([("migration_source/"+testfile, source_file_content)])
file_name = f"gpt_migrate/{testfile}.tests.py"
try:
with open(os.path.join(globals.targetdir, file_name), 'r') as file:
old_file_content = file.read()
except:
print("File not found: "+file_name+". Please ensure the file exists and try again. You can resume the debugging process with the `--step test` flag.")
raise typer.Exit()
debug_file_template = prompt_constructor(HIERARCHY, GUIDELINES, WRITE_CODE, DEBUG_TESTFILE, SINGLEFILE)
prompt = debug_file_template.format(error_message=error_message[-min(MAX_ERROR_MESSAGE_CHARACTERS, len(error_message)):],
file_name=file_name,
old_file_content=old_file_content,
relevant_files=relevant_files,
guidelines=globals.guidelines),
_, language, file_content = llm_write_file(prompt,
target_path=file_name,
waiting_message=f"Debugging {file_name}...",
success_message=f"Re-wrote {file_name} based on error message.",
globals=globals)
with open(os.path.join(globals.targetdir, file_name), 'r') as file:
new_file_content = file.read()
if new_file_content==old_file_content:
require_human_intervention(error_message,construct_relevant_files([(file_name, new_file_content)]),globals)
def require_human_intervention(error_message,relevant_files,globals):
human_intervention_template = prompt_constructor(HIERARCHY, GUIDELINES, WRITE_CODE, HUMAN_INTERVENTION, SINGLEFILE)
prompt = human_intervention_template.format(error_message=error_message[-min(MAX_ERROR_MESSAGE_CHARACTERS, len(error_message)):],
relevant_files=relevant_files,
target_directory_structure=build_directory_structure(globals.targetdir),
guidelines=globals.guidelines)
instructions = llm_run(prompt,
waiting_message=f"Writing instructions for how to proceed...",
success_message="",
globals=globals)
typer.echo(typer.style(f"GPT-Migrate is having some trouble debugging your app and requires human intervention. Below are instructions for how to fix your application.", fg=typer.colors.BLUE))
print(instructions)
typer.echo(typer.style(f"Once the fix is implemented, you can pick up from the testing phase using the `--step test` flag.", fg=typer.colors.BLUE))
raise typer.Exit()
================================================
FILE: gpt_migrate/steps/migrate.py
================================================
from utils import prompt_constructor, llm_write_file, llm_run, build_directory_structure, copy_files, write_to_memory, read_from_memory, file_exists_in_memory, convert_sigs_to_string
from config import HIERARCHY, GUIDELINES, WRITE_CODE, GET_EXTERNAL_DEPS, GET_INTERNAL_DEPS, ADD_DOCKER_REQUIREMENTS, REFINE_DOCKERFILE, WRITE_MIGRATION, SINGLEFILE, EXCLUDED_FILES, GET_FUNCTION_SIGNATURES
from typing import List
import os
import json
import typer
def get_function_signatures(targetfiles: List[str], globals):
''' Get the function signatures and a one-sentence summary for each function '''
all_sigs = []
for targetfile in targetfiles:
sigs_file_name = targetfile + "_sigs.json"
if file_exists_in_memory(sigs_file_name):
with open(os.path.join("memory", sigs_file_name), 'r') as f:
sigs = json.load(f)
all_sigs.extend(sigs)
else:
function_signatures_template = prompt_constructor(HIERARCHY, GUIDELINES, GET_FUNCTION_SIGNATURES)
targetfile_content = ""
with open(os.path.join(globals.targetdir, targetfile), 'r') as file:
targetfile_content = file.read()
prompt = function_signatures_template.format(targetlang=globals.targetlang,
sourcelang=globals.sourcelang,
targetfile_content=targetfile_content)
sigs = json.loads(llm_run(prompt,
waiting_message=f"Parsing function signatures for {targetfile}...",
success_message=None,
globals=globals))
all_sigs.extend(sigs)
with open(os.path.join('memory', sigs_file_name), 'w') as f:
json.dump(sigs, f)
return all_sigs
def get_dependencies(sourcefile, globals):
''' Get external and internal dependencies of source file '''
external_deps_prompt_template = prompt_constructor(HIERARCHY, GUIDELINES, GET_EXTERNAL_DEPS)
internal_deps_prompt_template = prompt_constructor(HIERARCHY, GUIDELINES, GET_INTERNAL_DEPS)
sourcefile_content = ""
with open(os.path.join(globals.sourcedir, sourcefile), 'r') as file:
sourcefile_content = file.read()
prompt = external_deps_prompt_template.format(targetlang=globals.targetlang,
sourcelang=globals.sourcelang,
sourcefile_content=sourcefile_content)
external_dependencies = llm_run(prompt,
waiting_message=f"Identifying external dependencies for {sourcefile}...",
success_message=None,
globals=globals)
external_deps_list = external_dependencies.split(',') if external_dependencies != "NONE" else []
write_to_memory("external_dependencies",external_deps_list)
prompt = internal_deps_prompt_template.format(targetlang=globals.targetlang,
sourcelang=globals.sourcelang,
sourcefile=sourcefile,
sourcefile_content=sourcefile_content,
source_directory_structure=globals.source_directory_structure)
internal_dependencies = llm_run(prompt,
waiting_message=f"Identifying internal dependencies for {sourcefile}...",
success_message=None,
globals=globals)
# Sanity checking internal dependencies to avoid infinite loops
if sourcefile in internal_dependencies:
typer.echo(typer.style(f"Warning: {sourcefile} seems to depend on itself. Automatically removing {sourcefile} from the list of internal dependencies.", fg=typer.colors.YELLOW))
internal_dependencies = internal_dependencies.replace(sourcefile, "")
internal_deps_list = [dep for dep in internal_dependencies.split(',') if dep] if internal_dependencies != "NONE" else []
return internal_deps_list, external_deps_list
def write_migration(sourcefile, external_deps_list, deps_per_file, globals) -> str:
''' Write migration file '''
sigs = get_function_signatures(deps_per_file, globals) if deps_per_file else []
write_migration_template = prompt_constructor(HIERARCHY, GUIDELINES, WRITE_CODE, WRITE_MIGRATION, SINGLEFILE)
sourcefile_content = ""
with open(os.path.join(globals.sourcedir, sourcefile), 'r') as file:
sourcefile_content = file.read()
prompt = write_migration_template.format(targetlang=globals.targetlang,
targetlang_function_signatures=convert_sigs_to_string(sigs),
sourcelang=globals.sourcelang,
sourcefile=sourcefile,
sourcefile_content=sourcefile_content,
external_deps=','.join(external_deps_list),
source_directory_structure=globals.source_directory_structure,
target_directory_structure=build_directory_structure(globals.targetdir),
guidelines=globals.guidelines)
return llm_write_file(prompt,
target_path=None,
waiting_message=f"Creating migration file for {sourcefile}...",
success_message=None,
globals=globals)[0]
def add_env_files(globals):
''' Copy all files recursively with included extensions from the source directory to the target directory in the same relative structure '''
copy_files(globals.sourcedir, globals.targetdir, excluded_files=EXCLUDED_FILES)
''' Add files required from the Dockerfile '''
add_docker_requirements_template = prompt_constructor(HIERARCHY, GUIDELINES, WRITE_CODE, ADD_DOCKER_REQUIREMENTS, SINGLEFILE)
dockerfile_content = ""
with open(os.path.join(globals.targetdir, 'Dockerfile'), 'r') as file:
dockerfile_content = file.read()
external_deps = read_from_memory("external_dependencies")
prompt = add_docker_requirements_template.format(dockerfile_content=dockerfile_content,
external_deps=external_deps,
target_directory_structure=build_directory_structure(globals.targetdir),
targetlang=globals.targetlang,
guidelines=globals.guidelines)
external_deps_name, _, external_deps_content = llm_write_file(prompt,
target_path=None,
waiting_message=f"Creating dependencies file required for the Docker environment...",
success_message=None,
globals=globals)
''' Refine Dockerfile '''
refine_dockerfile_template = prompt_constructor(HIERARCHY, GUIDELINES, WRITE_CODE, REFINE_DOCKERFILE, SINGLEFILE)
prompt = refine_dockerfile_template.format(dockerfile_content=dockerfile_content,
target_directory_structure=build_directory_structure(globals.targetdir),
external_deps_name=external_deps_name,
external_deps_content=external_deps_content,
guidelines=globals.guidelines)
llm_write_file(prompt,
target_path="Dockerfile",
waiting_message=f"Refining Dockerfile based on dependencies required for the Docker environment...",
success_message="Refined Dockerfile with dependencies required for the Docker environment.",
globals=globals)
================================================
FILE: gpt_migrate/steps/setup.py
================================================
from utils import prompt_constructor, llm_write_file
from config import HIERARCHY, GUIDELINES, WRITE_CODE, CREATE_DOCKER, SINGLEFILE
def create_environment(globals):
''' Create Dockerfile '''
docker_prompt_template = prompt_constructor(HIERARCHY, GUIDELINES, WRITE_CODE, CREATE_DOCKER, SINGLEFILE)
prompt = docker_prompt_template.format(targetlang=globals.targetlang,
sourcelang=globals.sourcelang,
sourceentry=globals.sourceentry,
guidelines=globals.guidelines)
llm_write_file(prompt,
target_path="Dockerfile",
waiting_message="Creating your environment...",
success_message=f"Created Docker environment for {globals.targetlang} project in directory '{globals.targetdir}'.",
globals=globals)
with open('memory/external_dependencies', 'w') as file:
file.write('')
================================================
FILE: gpt_migrate/steps/test.py
================================================
from utils import prompt_constructor, llm_write_file, construct_relevant_files, find_and_replace_file
from config import HIERARCHY, GUIDELINES, WRITE_CODE, CREATE_TESTS, SINGLEFILE
import subprocess
import typer
import os
import time as time
from yaspin import yaspin
from steps.debug import require_human_intervention
def run_dockerfile(globals):
try:
with yaspin(text="Spinning up Docker container...", spinner="dots") as spinner:
result = subprocess.run(["docker", "build", "-t", "gpt-migrate", globals.targetdir], stdout=subprocess.PIPE, stderr=subprocess.STDOUT, check=True, text=True)
subprocess.run(["docker", "rm", "-f", "gpt-migrate"])
process = subprocess.Popen(["docker", "run", "-d", "-p", "8080:8080", "--name", "gpt-migrate", "gpt-migrate"], stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True)
spinner.ok("✅ ")
success_text = typer.style("Your Docker image is now running. GPT-Migrate will now start testing, and you can independently test as well. The application is exposed on port 8080.", fg=typer.colors.GREEN)
typer.echo(success_text)
return "success"
except subprocess.CalledProcessError as e:
print("ERROR: ",e.output)
error_message = e.output
error_text = typer.style("Something isn't right with Docker. Please ensure Docker is running and take a look over the Dockerfile; there may be errors. Once these are resolved, you can resume your progress with the `--step test` flag.", fg=typer.colors.RED)
typer.echo(error_text)
# have typer ask if the user would like to use AI to fix it? If so, call function fix(). if not, raise typer.Exit()
if typer.confirm("Would you like GPT-Migrate to try to fix this?"):
return error_message
else:
dockerfile_content = ""
with open(os.path.join(globals.targetdir, 'Dockerfile'), 'r') as file:
dockerfile_content = file.read()
require_human_intervention(error_message,relevant_files=construct_relevant_files([("Dockerfile", dockerfile_content)]),globals=globals)
raise typer.Exit()
def create_tests(testfile,globals):
# Makedir gpt_migrate in targetdir if it doesn't exist
if not os.path.exists(os.path.join(globals.targetdir, 'gpt_migrate')):
os.makedirs(os.path.join(globals.targetdir, 'gpt_migrate'))
old_file_content = ""
with open(os.path.join(globals.sourcedir, testfile), 'r') as file:
old_file_content = file.read()
create_tests_template = prompt_constructor(HIERARCHY, GUIDELINES, WRITE_CODE, CREATE_TESTS, SINGLEFILE)
prompt = create_tests_template.format(targetport=globals.targetport,
old_file_content=old_file_content,
guidelines=globals.guidelines)
_, _, file_content = llm_write_file(prompt,
target_path=f"gpt_migrate/{testfile}.tests.py",
waiting_message="Creating tests file...",
success_message=f"Created {testfile}.tests.py file in directory gpt_migrate.",
globals=globals)
return f"{testfile}.tests.py"
def validate_tests(testfile,globals):
try:
with yaspin(text="Validating tests...", spinner="dots") as spinner:
# find all instances of globals.targetport in the testfile and replace with the port number globals.sourceport
find_and_replace_file(os.path.join(globals.targetdir, f"gpt_migrate/{testfile}"), str(globals.targetport), str(globals.sourceport))
time.sleep(0.3)
result = subprocess.run(["python3", os.path.join(globals.targetdir,f"gpt_migrate/{testfile}")], stdout=subprocess.PIPE, stderr=subprocess.STDOUT, check=True, text=True, timeout=15)
spinner.ok("✅ ")
print(result.stdout)
find_and_replace_file(os.path.join(globals.targetdir, f"gpt_migrate/{testfile}"), str(globals.sourceport), str(globals.targetport))
typer.echo(typer.style(f"Tests validated successfully on your source app.", fg=typer.colors.GREEN))
return "success"
except subprocess.CalledProcessError as e:
find_and_replace_file(os.path.join(globals.targetdir, f"gpt_migrate/{testfile}"), str(globals.sourceport), str(globals.targetport))
print("ERROR: ",e.output)
error_message = e.output
error_text = typer.style(f"Validating {testfile} against your existing service failed. Please take a look at the error message and try to resolve the issue. Once these are resolved, you can resume your progress with the `--step test` flag.", fg=typer.colors.RED)
typer.echo(error_text)
if typer.confirm("Would you like GPT-Migrate to try to fix this?"):
return error_message
else:
tests_content = ""
with open(os.path.join(globals.targetdir, f"gpt_migrate/{testfile}"), 'r') as file:
tests_content = file.read()
require_human_intervention(error_message,relevant_files=construct_relevant_files([(f"gpt_migrate/{testfile}", tests_content)]),globals=globals)
raise typer.Exit()
except subprocess.TimeoutExpired as e:
print(f"gpt_migrate/{testfile} timed out due to an unknown error and requires debugging.")
return f"gpt_migrate/{testfile} timed out due to an unknown error and requires debugging."
def run_test(testfile,globals):
try:
with yaspin(text="Running tests...", spinner="dots") as spinner:
time.sleep(0.3)
result = subprocess.run(["python3", os.path.join(globals.targetdir,f"gpt_migrate/{testfile}")], stdout=subprocess.PIPE, stderr=subprocess.STDOUT, check=True, text=True, timeout=15)
spinner.ok("✅ ")
print(result.stdout)
success_text = typer.style(f"Tests passed for {testfile}!", fg=typer.colors.GREEN)
typer.echo(success_text)
return "success"
except subprocess.CalledProcessError as e:
print("ERROR: ",e.output)
error_message = e.output
error_text = typer.style(f"One or more tests in {testfile} failed. Please take a look at the error message and try to resolve the issue. Once these are resolved, you can resume your progress with the `--step test` flag.", fg=typer.colors.RED)
typer.echo(error_text)
if typer.confirm("Would you like GPT-Migrate to try to fix this?"):
return error_message
else:
tests_content = ""
with open(os.path.join(globals.targetdir, f"gpt_migrate/{testfile}"), 'r') as file:
tests_content = file.read()
require_human_intervention(error_message,relevant_files=construct_relevant_files([(f"gpt_migrate/{testfile}", tests_content)]),globals=globals)
raise typer.Exit()
except subprocess.TimeoutExpired as e:
print(f"gpt_migrate/{testfile} timed out due to an unknown error and requires debugging.")
return f"gpt_migrate/{testfile} timed out due to an unknown error and requires debugging."
================================================
FILE: gpt_migrate/utils.py
================================================
import os
import typer
from yaspin import yaspin
from pathlib import Path
from collections import Counter
import fnmatch
import re
import shutil
from config import INCLUDED_EXTENSIONS, EXTENSION_TO_LANGUAGE
def detect_language(source_directory):
file_extensions = []
for filenames in os.walk(source_directory):
for filename in filenames[2]:
ext = filename.split('.')[-1]
file_extensions.append(ext)
extension_counts = Counter(file_extensions)
most_common_extension, _ = extension_counts.most_common(1)[0]
# Determine the language based on the most common file extension
language = EXTENSION_TO_LANGUAGE.get(most_common_extension, None)
return language
def prompt_constructor(*args):
prompt = ""
for arg in args:
with open(os.path.abspath(f'prompts/{arg}'), 'r') as file:
prompt += file.read().strip()
return prompt
def llm_run(prompt,waiting_message,success_message,globals):
output = ""
with yaspin(text=waiting_message, spinner="dots") as spinner:
output = globals.ai.run(prompt)
spinner.ok("✅ ")
if success_message:
success_text = typer.style(success_message, fg=typer.colors.GREEN)
typer.echo(success_text)
return output
def llm_write_file(prompt,target_path,waiting_message,success_message,globals):
file_content = ""
with yaspin(text=waiting_message, spinner="dots") as spinner:
file_name,language,file_content = globals.ai.write_code(prompt)[0]
spinner.ok("✅ ")
if file_name=="INSTRUCTIONS:":
return "INSTRUCTIONS:","",file_content
if target_path:
with open(os.path.join(globals.targetdir, target_path), 'w') as file:
file.write(file_content)
else:
with open(os.path.join(globals.targetdir, file_name), 'w') as file:
file.write(file_content)
if success_message:
success_text = typer.style(success_message, fg=typer.colors.GREEN)
typer.echo(success_text)
else:
success_text = typer.style(f"Created {file_name} at {globals.targetdir}", fg=typer.colors.GREEN)
typer.echo(success_text)
return file_name, language, file_content
def llm_write_files(prompt,target_path,waiting_message,success_message,globals):
file_content = ""
with yaspin(text=waiting_message, spinner="dots") as spinner:
results = globals.ai.write_code(prompt)
spinner.ok("✅ ")
for result in results:
file_name,language,file_content = result
if target_path:
with open(os.path.join(globals.targetdir, target_path), 'w') as file:
file.write(file_content)
else:
with open(os.path.join(globals.targetdir, file_name), 'w') as file:
file.write(file_content)
if not success_message:
success_text = typer.style(f"Created {file_name} at {globals.targetdir}", fg=typer.colors.GREEN)
typer.echo(success_text)
if success_message:
success_text = typer.style(success_message, fg=typer.colors.GREEN)
typer.echo(success_text)
return results
def load_templates_from_directory(directory_path):
templates = {}
for filename in os.listdir(directory_path):
with open(os.path.join(directory_path, filename), 'r') as file:
key = os.path.splitext(filename)[0]
templates[key] = file.read()
return templates
def parse_code_string(code_string):
sections = code_string.split('---')
pattern = re.compile(r'^(.+)\n```(.+?)\n(.*?)\n```', re.DOTALL)
code_triples = []
for section in sections:
match = pattern.match(section)
if match:
filename, language, code = match.groups()
code_triples.append((section.split("\n```")[0], language.strip(), code.strip()))
return code_triples
def read_gitignore(path):
gitignore_path = os.path.join(path, '.gitignore')
patterns = []
if os.path.exists(gitignore_path):
with open(gitignore_path, 'r') as file:
for line in file:
line = line.strip()
if line and not line.startswith('#'):
patterns.append(line)
return patterns
def is_ignored(entry_path, gitignore_patterns):
for pattern in gitignore_patterns:
if fnmatch.fnmatch(entry_path, pattern):
return True
return False
def build_directory_structure(path='.', indent='', is_last=True, parent_prefix='', is_root=True):
gitignore_patterns = read_gitignore(path) + [".gitignore", "*gpt_migrate/*"] if indent == '' else ["*gpt_migrate/*"]
base_name = os.path.basename(path)
if not base_name:
base_name = '.'
if indent == '':
prefix = '|-- ' if not is_root else ''
elif is_last:
prefix = parent_prefix + '└── '
else:
prefix = parent_prefix + '├── '
if os.path.isdir(path):
result = indent + prefix + base_name + '/\n' if not is_root else ''
else:
result = indent + prefix + base_name + '\n'
if os.path.isdir(path):
entries = os.listdir(path)
for index, entry in enumerate(entries):
entry_path = os.path.join(path, entry)
new_parent_prefix = ' ' if is_last else '│ '
if not is_ignored(entry_path, gitignore_patterns):
result += build_directory_structure(entry_path, indent + ' ', index == len(entries) - 1, parent_prefix + new_parent_prefix, is_root=False)
return result
def copy_files(sourcedir, targetdir, excluded_files=[]):
gitignore_patterns = read_gitignore(sourcedir) + [".gitignore"]
for item in os.listdir(sourcedir):
if os.path.isfile(os.path.join(sourcedir, item)):
if item.endswith(INCLUDED_EXTENSIONS) and item not in excluded_files:
if not is_ignored(item, gitignore_patterns):
os.makedirs(targetdir, exist_ok=True)
shutil.copy(os.path.join(sourcedir, item), targetdir)
typer.echo(typer.style(f"Copied {item} from {sourcedir} to {targetdir}", fg=typer.colors.GREEN))
else:
copy_files(os.path.join(sourcedir, item), os.path.join(targetdir, item))
def construct_relevant_files(files):
ret = ""
for file in files:
name = file[0]
content = file[1]
ret += name+":\n\n" + "```\n"+content+"\n```\n\n"
return ret
def file_exists_in_memory(filename):
file = Path('memory/' + filename)
return file.exists()
def convert_sigs_to_string(sigs):
sig_string = ""
for sig in sigs:
sig_string += sig["signature"] + "\n" + sig["description"] + "\n\n"
return sig_string
def write_to_memory(filename,content):
with open('memory/'+filename, 'a+') as file:
for item in content:
if item not in file.read().split("\n"):
file.write(item+'\n')
def read_from_memory(filename):
content = ""
with open('memory/'+filename, 'r') as file:
content = file.read()
return content
def find_and_replace_file(filepath,find,replace):
with open(filepath, 'r') as file:
testfile_content = file.read()
testfile_content = testfile_content.replace(find,replace)
with open(filepath, 'w') as file:
file.write(testfile_content)
================================================
FILE: pyproject.toml
================================================
[tool.poetry]
name = "gpt-migrate"
version = "0.1.0"
description = "Easily migrate your codebase from one framework or language to another."
authors = ["0xpayne"]
[tool.poetry.dependencies]
python = "^3.9"
typer = "^0.9.0"
langchain = "^0.0.238"
yaspin = "^2.3.0"
openai = "^0.27.8"
tree-sitter = "^0.20.1"
[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"
gitextract_fjcdwyzl/ ├── .gitignore ├── CODE_OF_CONDUCT.md ├── LICENSE ├── README.md ├── TERMS.md ├── benchmarks/ │ ├── flask-cpp/ │ │ └── source/ │ │ ├── .gitignore │ │ ├── app.py │ │ ├── db.py │ │ ├── requirements.txt │ │ └── storage/ │ │ └── items.json │ ├── flask-fastapi/ │ │ └── source/ │ │ ├── .gitignore │ │ ├── app.py │ │ ├── db.py │ │ ├── requirements.txt │ │ └── storage/ │ │ └── items.json │ ├── flask-nodejs/ │ │ └── source/ │ │ ├── .gitignore │ │ ├── app.py │ │ ├── db.py │ │ ├── requirements.txt │ │ └── storage/ │ │ └── items.json │ └── flask-rust/ │ └── source/ │ ├── .gitignore │ ├── app.py │ ├── db.py │ ├── requirements.txt │ └── storage/ │ └── items.json ├── gpt_migrate/ │ ├── ai.py │ ├── config.py │ ├── main.py │ ├── memory/ │ │ └── external_dependencies │ ├── parser.py │ ├── prompts/ │ │ ├── HIERARCHY │ │ ├── p1_guidelines/ │ │ │ └── guidelines │ │ ├── p2_actions/ │ │ │ └── write_code │ │ ├── p3_debug/ │ │ │ ├── create_file │ │ │ ├── debug_file │ │ │ ├── debug_target_docker │ │ │ ├── debug_testfile │ │ │ ├── human_intervention │ │ │ ├── identify_action │ │ │ ├── identify_file │ │ │ └── move_files │ │ ├── p3_migrate/ │ │ │ ├── 1_get_external_deps │ │ │ ├── 2_get_internal_deps │ │ │ ├── 3_write_migration │ │ │ ├── 4_add_docker_requirements │ │ │ ├── 5_refine_target_docker │ │ │ └── 6_get_function_signatures │ │ ├── p3_setup/ │ │ │ └── create_target_docker │ │ ├── p3_test/ │ │ │ └── create_tests │ │ └── p4_output_formats/ │ │ ├── file_debug │ │ ├── filenames │ │ ├── multi_file │ │ └── single_file │ ├── requirements.txt │ ├── steps/ │ │ ├── __init__.py │ │ ├── debug.py │ │ ├── migrate.py │ │ ├── setup.py │ │ └── test.py │ └── utils.py └── pyproject.toml
SYMBOL INDEX (65 symbols across 16 files)
FILE: benchmarks/flask-cpp/source/app.py
function hello_world (line 8) | def hello_world():
function get_grocery_items (line 12) | def get_grocery_items():
function add_grocery_item (line 21) | def add_grocery_item():
function delete_grocery_item (line 34) | def delete_grocery_item(item_id):
function hash_password (line 44) | def hash_password(password):
FILE: benchmarks/flask-cpp/source/db.py
function read_items (line 3) | def read_items():
function write_items (line 8) | def write_items(grocery_items):
FILE: benchmarks/flask-fastapi/source/app.py
function hello_world (line 8) | def hello_world():
function get_grocery_items (line 12) | def get_grocery_items():
function add_grocery_item (line 21) | def add_grocery_item():
function delete_grocery_item (line 34) | def delete_grocery_item(item_id):
function hash_password (line 44) | def hash_password(password):
FILE: benchmarks/flask-fastapi/source/db.py
function read_items (line 3) | def read_items():
function write_items (line 8) | def write_items(grocery_items):
FILE: benchmarks/flask-nodejs/source/app.py
function hello_world (line 8) | def hello_world():
function get_grocery_items (line 12) | def get_grocery_items():
function add_grocery_item (line 21) | def add_grocery_item():
function delete_grocery_item (line 34) | def delete_grocery_item(item_id):
function hash_password (line 44) | def hash_password(password):
FILE: benchmarks/flask-nodejs/source/db.py
function read_items (line 3) | def read_items():
function write_items (line 8) | def write_items(grocery_items):
FILE: benchmarks/flask-rust/source/app.py
function hello_world (line 8) | def hello_world():
function get_grocery_items (line 12) | def get_grocery_items():
function add_grocery_item (line 21) | def add_grocery_item():
function delete_grocery_item (line 34) | def delete_grocery_item(item_id):
function hash_password (line 44) | def hash_password(password):
FILE: benchmarks/flask-rust/source/db.py
function read_items (line 3) | def read_items():
function write_items (line 8) | def write_items(grocery_items):
FILE: gpt_migrate/ai.py
class AI (line 10) | class AI:
method __init__ (line 11) | def __init__(self, model='openrouter/openai/gpt-4-32k', temperature=0....
method write_code (line 21) | def write_code(self, prompt):
method run (line 36) | def run(self, prompt):
FILE: gpt_migrate/main.py
class Globals (line 16) | class Globals:
method __init__ (line 17) | def __init__(self, sourcedir, targetdir, sourcelang, targetlang, sourc...
function main (line 33) | def main(
FILE: gpt_migrate/parser.py
function decompose_file (line 10) | def decompose_file(file_path: str) -> Iterator[Node]:
FILE: gpt_migrate/steps/debug.py
function debug_error (line 7) | def debug_error(error_message,relevant_files,globals):
function debug_testfile (line 127) | def debug_testfile(error_message,testfile,globals):
function require_human_intervention (line 163) | def require_human_intervention(error_message,relevant_files,globals):
FILE: gpt_migrate/steps/migrate.py
function get_function_signatures (line 8) | def get_function_signatures(targetfiles: List[str], globals):
function get_dependencies (line 43) | def get_dependencies(sourcefile, globals):
function write_migration (line 86) | def write_migration(sourcefile, external_deps_list, deps_per_file, globa...
function add_env_files (line 114) | def add_env_files(globals):
FILE: gpt_migrate/steps/setup.py
function create_environment (line 4) | def create_environment(globals):
FILE: gpt_migrate/steps/test.py
function run_dockerfile (line 10) | def run_dockerfile(globals):
function create_tests (line 36) | def create_tests(testfile,globals):
function validate_tests (line 59) | def validate_tests(testfile,globals):
function run_test (line 90) | def run_test(testfile,globals):
FILE: gpt_migrate/utils.py
function detect_language (line 11) | def detect_language(source_directory):
function prompt_constructor (line 28) | def prompt_constructor(*args):
function llm_run (line 35) | def llm_run(prompt,waiting_message,success_message,globals):
function llm_write_file (line 48) | def llm_write_file(prompt,target_path,waiting_message,success_message,gl...
function llm_write_files (line 74) | def llm_write_files(prompt,target_path,waiting_message,success_message,g...
function load_templates_from_directory (line 100) | def load_templates_from_directory(directory_path):
function parse_code_string (line 108) | def parse_code_string(code_string):
function read_gitignore (line 123) | def read_gitignore(path):
function is_ignored (line 134) | def is_ignored(entry_path, gitignore_patterns):
function build_directory_structure (line 140) | def build_directory_structure(path='.', indent='', is_last=True, parent_...
function copy_files (line 170) | def copy_files(sourcedir, targetdir, excluded_files=[]):
function construct_relevant_files (line 182) | def construct_relevant_files(files):
function file_exists_in_memory (line 190) | def file_exists_in_memory(filename):
function convert_sigs_to_string (line 194) | def convert_sigs_to_string(sigs):
function write_to_memory (line 200) | def write_to_memory(filename,content):
function read_from_memory (line 206) | def read_from_memory(filename):
function find_and_replace_file (line 212) | def find_and_replace_file(filepath,find,replace):
Condensed preview — 61 files, each showing path, character count, and a content snippet. Download the .json file or copy for the full structured content (96K chars).
[
{
"path": ".gitignore",
"chars": 37,
"preview": "**/target/\n.DS_Store\n**/__pycache__/\n"
},
{
"path": "CODE_OF_CONDUCT.md",
"chars": 1814,
"preview": "# Code of Conduct for GPT-Migrate\n\n## 1. Purpose\n\nThe purpose of this Code of Conduct is to provide guidelines for contr"
},
{
"path": "LICENSE",
"chars": 1066,
"preview": "MIT License\n\nCopyright (c) 2023 Josh Payne\n\nPermission is hereby granted, free of charge, to any person obtaining a copy"
},
{
"path": "README.md",
"chars": 10044,
"preview": "<div align=\"center\">\n\n# ◐ GPT-Migrate ◑\n\n**Easily migrate your codebase from one framework or language to "
},
{
"path": "TERMS.md",
"chars": 2049,
"preview": "# Terms of Use\n\nBy using GPT-Migrate you are aware of and agree to the below Terms of Use.\n\nGPT-Migrate is an experiment"
},
{
"path": "benchmarks/flask-cpp/source/.gitignore",
"chars": 25,
"preview": "ignore_me\n**/__pycache__/"
},
{
"path": "benchmarks/flask-cpp/source/app.py",
"chars": 1582,
"preview": "from flask import Flask, request, jsonify\nfrom bcrypt import hashpw, gensalt\nfrom db import read_items, write_items\n\napp"
},
{
"path": "benchmarks/flask-cpp/source/db.py",
"chars": 251,
"preview": "import json\n\ndef read_items():\n with open('storage/items.json') as f:\n grocery_items = json.load(f)\n return"
},
{
"path": "benchmarks/flask-cpp/source/requirements.txt",
"chars": 12,
"preview": "flask\nbcrypt"
},
{
"path": "benchmarks/flask-cpp/source/storage/items.json",
"chars": 124,
"preview": "[{\"id\": 1, \"name\": \"Milk\", \"price\": 2.5}, {\"id\": 2, \"name\": \"Eggs\", \"price\": 3.0}, {\"id\": 3, \"name\": \"Bread\", \"price\": 1"
},
{
"path": "benchmarks/flask-fastapi/source/.gitignore",
"chars": 25,
"preview": "ignore_me\n**/__pycache__/"
},
{
"path": "benchmarks/flask-fastapi/source/app.py",
"chars": 1582,
"preview": "from flask import Flask, request, jsonify\nfrom bcrypt import hashpw, gensalt\nfrom db import read_items, write_items\n\napp"
},
{
"path": "benchmarks/flask-fastapi/source/db.py",
"chars": 251,
"preview": "import json\n\ndef read_items():\n with open('storage/items.json') as f:\n grocery_items = json.load(f)\n return"
},
{
"path": "benchmarks/flask-fastapi/source/requirements.txt",
"chars": 12,
"preview": "flask\nbcrypt"
},
{
"path": "benchmarks/flask-fastapi/source/storage/items.json",
"chars": 124,
"preview": "[{\"id\": 1, \"name\": \"Milk\", \"price\": 2.5}, {\"id\": 2, \"name\": \"Eggs\", \"price\": 3.0}, {\"id\": 3, \"name\": \"Bread\", \"price\": 1"
},
{
"path": "benchmarks/flask-nodejs/source/.gitignore",
"chars": 25,
"preview": "ignore_me\n**/__pycache__/"
},
{
"path": "benchmarks/flask-nodejs/source/app.py",
"chars": 1582,
"preview": "from flask import Flask, request, jsonify\nfrom bcrypt import hashpw, gensalt\nfrom db import read_items, write_items\n\napp"
},
{
"path": "benchmarks/flask-nodejs/source/db.py",
"chars": 251,
"preview": "import json\n\ndef read_items():\n with open('storage/items.json') as f:\n grocery_items = json.load(f)\n return"
},
{
"path": "benchmarks/flask-nodejs/source/requirements.txt",
"chars": 12,
"preview": "flask\nbcrypt"
},
{
"path": "benchmarks/flask-nodejs/source/storage/items.json",
"chars": 124,
"preview": "[{\"id\": 1, \"name\": \"Milk\", \"price\": 2.5}, {\"id\": 2, \"name\": \"Eggs\", \"price\": 3.0}, {\"id\": 3, \"name\": \"Bread\", \"price\": 1"
},
{
"path": "benchmarks/flask-rust/source/.gitignore",
"chars": 25,
"preview": "ignore_me\n**/__pycache__/"
},
{
"path": "benchmarks/flask-rust/source/app.py",
"chars": 1582,
"preview": "from flask import Flask, request, jsonify\nfrom bcrypt import hashpw, gensalt\nfrom db import read_items, write_items\n\napp"
},
{
"path": "benchmarks/flask-rust/source/db.py",
"chars": 251,
"preview": "import json\n\ndef read_items():\n with open('storage/items.json') as f:\n grocery_items = json.load(f)\n return"
},
{
"path": "benchmarks/flask-rust/source/requirements.txt",
"chars": 12,
"preview": "flask\nbcrypt"
},
{
"path": "benchmarks/flask-rust/source/storage/items.json",
"chars": 124,
"preview": "[{\"id\": 1, \"name\": \"Milk\", \"price\": 2.5}, {\"id\": 2, \"name\": \"Eggs\", \"price\": 3.0}, {\"id\": 3, \"name\": \"Bread\", \"price\": 1"
},
{
"path": "gpt_migrate/ai.py",
"chars": 1745,
"preview": "from langchain.chat_models import ChatOpenAI\nfrom config import OPENAI_API_KEY\nimport os\nimport openai\nfrom utils import"
},
{
"path": "gpt_migrate/config.py",
"chars": 3342,
"preview": "import os\n\n'''\nEnvironment variables\n'''\nOPENAI_API_KEY = os.environ.get(\"OPENAI_API_KEY\")\n\n'''\nGlobal variables\n'''\nMAX"
},
{
"path": "gpt_migrate/main.py",
"chars": 6604,
"preview": "import typer\nfrom utils import detect_language, build_directory_structure\nimport os\nimport time as time\nfrom collections"
},
{
"path": "gpt_migrate/memory/external_dependencies",
"chars": 51,
"preview": "rocket\nserde\nserde_json\nbcrypt\nrusqlite\nserde_json\n"
},
{
"path": "gpt_migrate/parser.py",
"chars": 1855,
"preview": "import subprocess\nimport typer\nfrom yaspin import yaspin\nfrom pathlib import Path\nfrom tree_sitter import Language, Pars"
},
{
"path": "gpt_migrate/prompts/HIERARCHY",
"chars": 464,
"preview": "The following prompt is a composition of prompt sections, each with different preference levels. Higher preference level"
},
{
"path": "gpt_migrate/prompts/p1_guidelines/guidelines",
"chars": 684,
"preview": "\\n\\n PREFERENCE LEVEL 1\n\nHere are the guidelines for this prompt:\n\n1. Follow the output instructions precisely and do no"
},
{
"path": "gpt_migrate/prompts/p2_actions/write_code",
"chars": 840,
"preview": "\\n\\n PREFERENCE LEVEL 2\n\nYou are a pragmatic principal engineer at Google. You are about to get instructions for code to"
},
{
"path": "gpt_migrate/prompts/p3_debug/create_file",
"chars": 550,
"preview": "\\n\\n PREFERENCE LEVEL 3\n\nYou are a principal software engineer at Google. The current app is having trouble running. Con"
},
{
"path": "gpt_migrate/prompts/p3_debug/debug_file",
"chars": 524,
"preview": "\\n\\n PREFERENCE LEVEL 3\n\nYou are a principal software engineer at Google. We are doing a migration to {targetlang} from "
},
{
"path": "gpt_migrate/prompts/p3_debug/debug_target_docker",
"chars": 516,
"preview": "\\n\\n PREFERENCE LEVEL 3\n\nYou are a principal software engineer at Google with particular expertise in Docker environment"
},
{
"path": "gpt_migrate/prompts/p3_debug/debug_testfile",
"chars": 646,
"preview": "\\n\\n PREFERENCE LEVEL 3\n\nYou are a principal software engineer at Google. We are writing {file_name} using unittest agai"
},
{
"path": "gpt_migrate/prompts/p3_debug/human_intervention",
"chars": 413,
"preview": "\\n\\n PREFERENCE LEVEL 3\n\nYou are a principal software engineer at Google who is pair programming with a partner. You've "
},
{
"path": "gpt_migrate/prompts/p3_debug/identify_action",
"chars": 918,
"preview": "\\n\\n PREFERENCE LEVEL 3\n\nYou are a principal software engineer at Google pair programming with a partner. The current Do"
},
{
"path": "gpt_migrate/prompts/p3_debug/identify_file",
"chars": 1029,
"preview": "\\n\\n PREFERENCE LEVEL 3\n\nYou are a principal software engineer at Google. The current Docker app is having trouble runni"
},
{
"path": "gpt_migrate/prompts/p3_debug/move_files",
"chars": 858,
"preview": "\\n\\n PREFERENCE LEVEL 3\n\nYou are a principal software engineer at Google. The current app is having trouble running. Con"
},
{
"path": "gpt_migrate/prompts/p3_migrate/1_get_external_deps",
"chars": 731,
"preview": "\\n\\n PREFERENCE LEVEL 3\n\nYou are a principal software engineer at Google with particular expertise migrating codebases f"
},
{
"path": "gpt_migrate/prompts/p3_migrate/2_get_internal_deps",
"chars": 1139,
"preview": "\\n\\n PREFERENCE LEVEL 3\n\nYou are a principal software engineer at Google with particular expertise migrating codebases f"
},
{
"path": "gpt_migrate/prompts/p3_migrate/3_write_migration",
"chars": 1665,
"preview": "\\n\\n PREFERENCE LEVEL 3\n\nYou are a principal software engineer at Google with particular expertise migrating codebases f"
},
{
"path": "gpt_migrate/prompts/p3_migrate/4_add_docker_requirements",
"chars": 597,
"preview": "\\n\\n PREFERENCE LEVEL 3\n\nYou are a principal software engineer at Google with particular expertise in Docker environment"
},
{
"path": "gpt_migrate/prompts/p3_migrate/5_refine_target_docker",
"chars": 515,
"preview": "\\n\\n PREFERENCE LEVEL 3\n\nYou are a principal software engineer at Google with particular expertise in Docker environment"
},
{
"path": "gpt_migrate/prompts/p3_migrate/6_get_function_signatures",
"chars": 1340,
"preview": "\\n\\n PREFERENCE LEVEL 3\n\nYou are a principal software engineer at Google with particular expertise migrating codebases f"
},
{
"path": "gpt_migrate/prompts/p3_setup/create_target_docker",
"chars": 555,
"preview": "\\n\\n PREFERENCE LEVEL 3\n\nPlease create a Dockerfile for a generic app in the following framework: {targetlang}. This new"
},
{
"path": "gpt_migrate/prompts/p3_test/create_tests",
"chars": 936,
"preview": "\\n\\n PREFERENCE LEVEL 3\n\nYou are a principal software engineer at Google. You're responsible for creating a set of tests"
},
{
"path": "gpt_migrate/prompts/p4_output_formats/file_debug",
"chars": 853,
"preview": "\\n\\n PREFERENCE LEVEL 4\n\nWe will be using the output you provide as-is to create new files OR provide natural language i"
},
{
"path": "gpt_migrate/prompts/p4_output_formats/filenames",
"chars": 349,
"preview": "\\n\\n PREFERENCE LEVEL 4\n\nPlease respond only with a comma-separated list of items: \n\nfile.ext,dir1/file2.ext,file3.ext.."
},
{
"path": "gpt_migrate/prompts/p4_output_formats/multi_file",
"chars": 421,
"preview": "\\n\\n PREFERENCE LEVEL 4\n\nWe will be using the output you provide as-is to create new files, so please be precise and do "
},
{
"path": "gpt_migrate/prompts/p4_output_formats/single_file",
"chars": 378,
"preview": "\\n\\n PREFERENCE LEVEL 4\n\nWe will be using the output you provide as-is to create new files, so please be precise and do "
},
{
"path": "gpt_migrate/requirements.txt",
"chars": 76,
"preview": "typer\nlangchain\nyaspin\nopenai\ntree_sitter\nlitellm==0.1.213\npydantic==1.10.8\n"
},
{
"path": "gpt_migrate/steps/__init__.py",
"chars": 0,
"preview": ""
},
{
"path": "gpt_migrate/steps/debug.py",
"chars": 10824,
"preview": "from utils import prompt_constructor, llm_write_file, llm_run, build_directory_structure, construct_relevant_files\nfrom "
},
{
"path": "gpt_migrate/steps/migrate.py",
"chars": 8254,
"preview": "from utils import prompt_constructor, llm_write_file, llm_run, build_directory_structure, copy_files, write_to_memory, r"
},
{
"path": "gpt_migrate/steps/setup.py",
"chars": 1016,
"preview": "from utils import prompt_constructor, llm_write_file\nfrom config import HIERARCHY, GUIDELINES, WRITE_CODE, CREATE_DOCKER"
},
{
"path": "gpt_migrate/steps/test.py",
"chars": 7213,
"preview": "from utils import prompt_constructor, llm_write_file, construct_relevant_files, find_and_replace_file\nfrom config import"
},
{
"path": "gpt_migrate/utils.py",
"chars": 7397,
"preview": "import os\nimport typer\nfrom yaspin import yaspin\nfrom pathlib import Path\nfrom collections import Counter\nimport fnmatch"
},
{
"path": "pyproject.toml",
"chars": 399,
"preview": "[tool.poetry]\nname = \"gpt-migrate\"\nversion = \"0.1.0\"\ndescription = \"Easily migrate your codebase from one framework or l"
}
]
About this extraction
This page contains the full source code of the joshpxyne/gpt-migrate GitHub repository, extracted and formatted as plain text for AI agents and large language models (LLMs). The extraction includes 61 files (86.6 KB), approximately 21.3k tokens, and a symbol index with 65 extracted functions, classes, methods, constants, and types. Use this with OpenClaw, Claude, ChatGPT, Cursor, Windsurf, or any other AI tool that accepts text input. You can copy the full output to your clipboard or download it as a .txt file.
Extracted by GitExtract — free GitHub repo to text converter for AI. Built by Nikandr Surkov.