Repository: joshpxyne/gpt-migrate Branch: main Commit: 262cabb5e8da Files: 61 Total size: 86.6 KB Directory structure: gitextract_fjcdwyzl/ ├── .gitignore ├── CODE_OF_CONDUCT.md ├── LICENSE ├── README.md ├── TERMS.md ├── benchmarks/ │ ├── flask-cpp/ │ │ └── source/ │ │ ├── .gitignore │ │ ├── app.py │ │ ├── db.py │ │ ├── requirements.txt │ │ └── storage/ │ │ └── items.json │ ├── flask-fastapi/ │ │ └── source/ │ │ ├── .gitignore │ │ ├── app.py │ │ ├── db.py │ │ ├── requirements.txt │ │ └── storage/ │ │ └── items.json │ ├── flask-nodejs/ │ │ └── source/ │ │ ├── .gitignore │ │ ├── app.py │ │ ├── db.py │ │ ├── requirements.txt │ │ └── storage/ │ │ └── items.json │ └── flask-rust/ │ └── source/ │ ├── .gitignore │ ├── app.py │ ├── db.py │ ├── requirements.txt │ └── storage/ │ └── items.json ├── gpt_migrate/ │ ├── ai.py │ ├── config.py │ ├── main.py │ ├── memory/ │ │ └── external_dependencies │ ├── parser.py │ ├── prompts/ │ │ ├── HIERARCHY │ │ ├── p1_guidelines/ │ │ │ └── guidelines │ │ ├── p2_actions/ │ │ │ └── write_code │ │ ├── p3_debug/ │ │ │ ├── create_file │ │ │ ├── debug_file │ │ │ ├── debug_target_docker │ │ │ ├── debug_testfile │ │ │ ├── human_intervention │ │ │ ├── identify_action │ │ │ ├── identify_file │ │ │ └── move_files │ │ ├── p3_migrate/ │ │ │ ├── 1_get_external_deps │ │ │ ├── 2_get_internal_deps │ │ │ ├── 3_write_migration │ │ │ ├── 4_add_docker_requirements │ │ │ ├── 5_refine_target_docker │ │ │ └── 6_get_function_signatures │ │ ├── p3_setup/ │ │ │ └── create_target_docker │ │ ├── p3_test/ │ │ │ └── create_tests │ │ └── p4_output_formats/ │ │ ├── file_debug │ │ ├── filenames │ │ ├── multi_file │ │ └── single_file │ ├── requirements.txt │ ├── steps/ │ │ ├── __init__.py │ │ ├── debug.py │ │ ├── migrate.py │ │ ├── setup.py │ │ └── test.py │ └── utils.py └── pyproject.toml ================================================ FILE CONTENTS ================================================ ================================================ FILE: .gitignore ================================================ **/target/ .DS_Store **/__pycache__/ ================================================ FILE: CODE_OF_CONDUCT.md ================================================ # Code of Conduct for GPT-Migrate ## 1. Purpose The purpose of this Code of Conduct is to provide guidelines for contributors to the GPT-Migrate project on GitHub. We aim to create a positive and inclusive environment where all participants can contribute and collaborate effectively. By participating in this project, you agree to abide by this Code of Conduct. ## 2. Scope This Code of Conduct applies to all contributors, maintainers, and users of the GPT-Migrate project. It extends to all project spaces, including but not limited to issues, pull requests, code reviews, comments, and other forms of communication within the project. ## 3. Our Standards We encourage the following behavior: * Being respectful and considerate to others * Actively seeking diverse perspectives * Providing constructive feedback and assistance * Demonstrating empathy and understanding We discourage the following behavior: * Harassment or discrimination of any kind * Disrespectful, offensive, or inappropriate language or content * Personal attacks or insults * Unwarranted criticism or negativity ## 4. Reporting and Enforcement If you witness or experience any violations of this Code of Conduct, please report them to the project maintainers by email or other appropriate means. The maintainers will investigate and take appropriate action, which may include warnings, temporary or permanent bans, or other measures as necessary. Maintainers are responsible for ensuring compliance with this Code of Conduct and may take action to address any violations. ## 5. Acknowledgements This Code of Conduct is adapted from the [Contributor Covenant](https://www.contributor-covenant.org/version/2/0/code_of_conduct.html). ## 6. Contact If you have any questions or concerns, please contact the project maintainers. ================================================ FILE: LICENSE ================================================ MIT License Copyright (c) 2023 Josh Payne Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. ================================================ FILE: README.md ================================================
# ◐   GPT-Migrate   ◑ **Easily migrate your codebase from one framework or language to another.**

Github License GitHub Repo stars


If you've ever faced the pain of migrating a codebase to a new framework or language, this project is for you. https://user-images.githubusercontent.com/25165841/250232917-bcc99ce8-99b7-4e3d-a653-f89e163ed825.mp4 Migration is a costly, tedious, and non-trivial problem. Do not trust the current version blindly and please use responsibly. Please also be aware that costs can add up quickly as GPT-Migrate is designed to write (and potentially re-write) the entirety of a codebase. However, with the collective brilliance of the OSS community and the current state of LLMs, it is also a very tractable problem. ## ⚡️ Usage 1. Install Docker and ensure that it's running. It's also recommended that you use at least GPT-4, preferably GPT-4-32k. ## 📦 Installation using Poetry 1. Install Poetry by following the instructions on the [official Poetry website](https://python-poetry.org/docs/#installation). 2. Once Poetry is installed, navigate to the project directory and install the project dependencies using the following command: ```bash poetry install ``` This will create a virtual environment and install all the necessary dependencies in that environment. 2. Set your [OpenRouter API key](https://openrouter.ai/docs#api-keys) (default) and/or your [OpenAI API key](https://platform.openai.com/account/api-keys) (to use the OpenAI API directly...in this case, set --model to `gpt-4-32k` or your desired model) and install the python requirements: `export OPENROUTER_API_KEY=` `export OPENAI_API_KEY=` `pip install -r requirements.txt` 3. Run the main script with the target language you want to migrate to: `python main.py --targetlang nodejs` 4. (Optional) If you'd like GPT-Migrate to validate the unit tests it creates against your app before it tests the migrated app with them, please have your existing app exposed and use the `--sourceport` flag. For executing this against the benchmark, open a separate terminal, navigate to the `benchmarks/language-pair/source` directory, and run `python app.py` after installing the requirements. It will expose on port 5000. Use this with the `--sourceport` flag. By default, this script will execute the flask-nodejs benchmark. You can specify the language, source directory, and many other things using the options guide below. ## 💡 Options You can customize the behavior of GPT-Migrate by passing the following options to the `main.py` script: - `--model`: The Large Language Model to be used. Default is `"gpt-4-32k"`. - `--temperature`: Temperature setting for the AI model. Default is `0`. - `--sourcedir`: Source directory containing the code to be migrated. Default is `"../benchmarks/flask-nodejs/source"`. - `--sourcelang`: Source language or framework of the code to be migrated. No default value. - `--sourceentry`: Entrypoint filename relative to the source directory. For instance, this could be an `app.py` or `main.py` file for Python. Default is `"app.py"`. - `--targetdir`: Directory where the migrated code will live. Default is `"../benchmarks/flask-nodejs/target"`. - `--targetlang`: Target language or framework for migration. Default is `"nodejs"`. - `--operating_system`: Operating system for the Dockerfile. Common options are `'linux'` or `'windows'`. Default is `'linux'`. - `--testfiles`: Comma-separated list of files that have functions to be tested. For instance, this could be an `app.py` or `main.py` file for a Python app where your REST endpoints are. Include the full relative path. Default is `"app.py"`. - `--sourceport`: (Optional) Port for testing the unit tests file against the original app. No default value. If not included, GPT-Migrate will not attempt to test the unit tests against your original app. - `--targetport`: Port for testing the unit tests file against the migrated app. Default is `8080`. - `--guidelines`: Stylistic or small functional guidelines that you'd like to be followed during the migration. For instance, "Use tabs, not spaces". Default is an empty string. - `--step`: Step to run. Options are `'setup'`, `'migrate'`, `'test'`, `'all'`. Default is `'all'`. For example, to migrate a Python codebase to Node.js, you might run: ```bash python main.py --sourcedir /path/to/my-python-app --sourceentry app.py --targetdir /path/to/my-nodejs-app --targetlang nodejs ``` This will take the Python code in `./my-python-app`, migrate it to Node.js, and write the resulting code to `./my-nodejs-app`. #### GPT-assisted debugging https://user-images.githubusercontent.com/25165841/250233075-eff1a535-f40e-42e4-914c-042c69ba9195.mp4 ## 🤖 How it Works For migrating a repo from `--sourcelang` to `--targetlang`... 1. GPT-Migrate first creates a Docker environment for `--targetlang`, which is either passed in or assessed automatically by GPT-Migrate. 2. It evaluates your existing code recursively to identify 3rd-party `--sourcelang` dependencies and selects corresponding `--targetlang` dependencies. 3. It recursively rebuilds new `--targetlang` code from your existing code starting from your designated `--sourceentry` file. This step can be started from with the `--step migrate` option. 4. It spins up the Docker environment with the new codebase, exposing it on `--targetport` and iteratively debugging as needed. 5. It develops unit tests using Python's unittest framework, and optionally tests these against your existing app if it's running and exposed on `--sourceport`, iteratively debugging as needed. This step can be started from with the `--step test` option. 6. It tests the new code on `--targetport` against these unit tests. 7. It iteratively debugs the code for for you with context from logs, error messages, relevant files, and directory structure. It does so by choosing one or more actions (move, create, or edit files) then executing them. If it wants to execute any sort of shell script (moving files around), it will first ask for clearance. Finally, if at any point it gets stuck or the user ends the debugging loop, it will output directions for the user to follow to move to the next step of the migration. 8. The new codebase is completed and exists in `--targetdir`. ### 📝 Prompt Design Subprompts are organized in the following fashion: - `HIERARCHY`: this defines the notion of preferences. There are 4 levels of preference, and each level prioritized more highly than the previous one. - `p1`: Preference Level 1. These are the most general prompts, and consist of broad guidelines. - `p2`: Preference Level 2. These are more specific prompts, and consist of guidelines for certain types of actions (e.g., best practices and philosophies for writing code). - `p3`: Preference Level 3. These are even more specific prompts, and consist of directions for specific actions (e.g., creating a certain file, debugging, writing tests). - `p4`: Preference Level 4. These are the most specific prompts, and consist of formatting for output. Prompts are a combination of subprompts. This concept of tagging and composability can be extended to other properties as well to make prompts even more robust. This is an area we're highly interested in actively exploring. In this repo, the `prompt_constructor()` function takes in one or more subprompts and yields a string which may be formatted with variables, for example with `GUIDELINES` being a `p1`, `WRITE_CODE` being a `p2` etc: ```python prompt = prompt_constructor(HIERARCHY, GUIDELINES, WRITE_CODE, DEBUG_TESTFILE, SINGLEFILE).format(targetlang=targetlang,buggyfile=buggyfile) ``` ## 📈 Performance GPT-Migrate is currently in development alpha and is not yet ready for production use. For instance, on the relatively simple benchmarks, it gets through "easy" languages like python or javascript without a hitch ~50% of the time, and cannot get through more complex languages like C++ or Rust without some human assistance. ## ✅ Benchmarks We're actively looking to build up a robust benchmark repository. If you have a codebase that you'd like to contribute, please open a PR! The current benchmarks were built from scratch: REST API apps which have a few endpoints and dependency files. ## 🧗 Roadmap Below are improvements on the to-do list. If you'd like to knock any of these or others out, please submit a PR :) #### High urgency - Add logic for model input size limiting based on the window size. See issue [#2](https://github.com/0xpayne/gpt-migrate/issues/2). #### Med urgency - Add unit tests to the entire project for better reliability and CI/CD - Add more benchmark examples, especially larger repos - Add functionality to let the LLM request access to dependency functions in other files as it debugs - Add support for other LLMs #### Low urgency - Enable internet search requests as the model debugs - Identify and compile language-specific issues + solve for them ## 📣 Call to Action We're looking for talented contributors. Whether you have a particular passion about a specific language or framework, want to help in creating a more robust test suite, or generally have interesting ideas on how to make this better, we'd love to have you! ## 🛠 Expert-Assisted Migration Due to the inflow of requests, we've decided to create a standardized process for helping people with their migrations. If you're a company that needs help with a big migration or an expert that is willing to help with them, please visit the following website: [https://gpt-migrate.com/](https://gpt-migrate.com/) ## Join the conversation on [Twitter](https://twitter.com/joshpxyne/status/1675254164165910528)! ================================================ FILE: TERMS.md ================================================ # Terms of Use By using GPT-Migrate you are aware of and agree to the below Terms of Use. GPT-Migrate is an experimental application and is provided "as-is" without any warranty, express or implied. By using this software, you agree to assume all risks associated with its use, including but not limited to data loss, system failure, or any other issues that may arise. The developers and contributors of this project do not accept any responsibility or liability for any losses, damages, or other consequences that may occur as a result of using this software. You are solely responsible for any decisions and actions taken based on the information provided by GPT-Migrate. Please note that the use of large language models can be expensive. By utilizing this project, you acknowledge that you are responsible for monitoring and managing your own token usage and the associated costs. It is highly recommended to check your usage regularly and set up any necessary limits or alerts to prevent unexpected charges. This is especially true of GPT-Migrate, which will write and potentially re-write the entirety of a codebase. As an autonomous experiment, GPT-Migrate may generate code or take actions that are not in line with real-world business practices or legal requirements. It is your responsibility to ensure that any actions or decisions made by the generated code comply with all applicable laws, regulations, and ethical standards. The developers and contributors of this project shall not be held responsible for any consequences arising from the use of this software. By using GPT-Migrate, you agree to indemnify, defend, and hold harmless the developers, contributors, and any affiliated parties from and against any and all claims, damages, losses, liabilities, costs, and expenses (including reasonable attorneys' fees) arising from your use of this software or your violation of these terms. More information about OpenAI's terms of use, which apply to GPT-Migrate, can be found [here](https://openai.com/policies/terms-of-use). ================================================ FILE: benchmarks/flask-cpp/source/.gitignore ================================================ ignore_me **/__pycache__/ ================================================ FILE: benchmarks/flask-cpp/source/app.py ================================================ from flask import Flask, request, jsonify from bcrypt import hashpw, gensalt from db import read_items, write_items app = Flask(__name__) @app.route('/', methods=['GET']) def hello_world(): return "Hello World!" @app.route('/grocery_items', methods=['GET']) def get_grocery_items(): try: grocery_items = read_items() items = [{"id": item["id"], "name": item["name"], "price": item["price"]} for item in grocery_items] return jsonify(items) except Exception as e: return e, 500 @app.route('/grocery_items', methods=['POST']) def add_grocery_item(): try: new_item = request.json print(new_item["id"],new_item,flush=True) grocery_items = read_items() if new_item not in grocery_items: grocery_items.append(new_item) write_items(grocery_items) return "Successfully added item", 201 except Exception as e: return e, 500 @app.route('/grocery_items/', methods=['DELETE']) def delete_grocery_item(item_id): try: grocery_items = read_items() grocery_items = [item for item in grocery_items if item["id"] != item_id] write_items(grocery_items) return "Successfully deleted item", 200 except Exception as e: return e, 500 @app.route('/hashpassword/', methods=['GET']) def hash_password(password): try: return hashpw(password.encode('utf-8'), gensalt()).decode('utf-8') except Exception as e: return e, 500 if __name__ == '__main__': app.run(debug=True) ================================================ FILE: benchmarks/flask-cpp/source/db.py ================================================ import json def read_items(): with open('storage/items.json') as f: grocery_items = json.load(f) return grocery_items def write_items(grocery_items): with open('storage/items.json', 'w') as f: json.dump(grocery_items, f) ================================================ FILE: benchmarks/flask-cpp/source/requirements.txt ================================================ flask bcrypt ================================================ FILE: benchmarks/flask-cpp/source/storage/items.json ================================================ [{"id": 1, "name": "Milk", "price": 2.5}, {"id": 2, "name": "Eggs", "price": 3.0}, {"id": 3, "name": "Bread", "price": 1.5}] ================================================ FILE: benchmarks/flask-fastapi/source/.gitignore ================================================ ignore_me **/__pycache__/ ================================================ FILE: benchmarks/flask-fastapi/source/app.py ================================================ from flask import Flask, request, jsonify from bcrypt import hashpw, gensalt from db import read_items, write_items app = Flask(__name__) @app.route('/', methods=['GET']) def hello_world(): return "Hello World!" @app.route('/grocery_items', methods=['GET']) def get_grocery_items(): try: grocery_items = read_items() items = [{"id": item["id"], "name": item["name"], "price": item["price"]} for item in grocery_items] return jsonify(items) except Exception as e: return e, 500 @app.route('/grocery_items', methods=['POST']) def add_grocery_item(): try: new_item = request.json print(new_item["id"],new_item,flush=True) grocery_items = read_items() if new_item not in grocery_items: grocery_items.append(new_item) write_items(grocery_items) return "Successfully added item", 201 except Exception as e: return e, 500 @app.route('/grocery_items/', methods=['DELETE']) def delete_grocery_item(item_id): try: grocery_items = read_items() grocery_items = [item for item in grocery_items if item["id"] != item_id] write_items(grocery_items) return "Successfully deleted item", 200 except Exception as e: return e, 500 @app.route('/hashpassword/', methods=['GET']) def hash_password(password): try: return hashpw(password.encode('utf-8'), gensalt()).decode('utf-8') except Exception as e: return e, 500 if __name__ == '__main__': app.run(debug=True) ================================================ FILE: benchmarks/flask-fastapi/source/db.py ================================================ import json def read_items(): with open('storage/items.json') as f: grocery_items = json.load(f) return grocery_items def write_items(grocery_items): with open('storage/items.json', 'w') as f: json.dump(grocery_items, f) ================================================ FILE: benchmarks/flask-fastapi/source/requirements.txt ================================================ flask bcrypt ================================================ FILE: benchmarks/flask-fastapi/source/storage/items.json ================================================ [{"id": 1, "name": "Milk", "price": 2.5}, {"id": 2, "name": "Eggs", "price": 3.0}, {"id": 3, "name": "Bread", "price": 1.5}] ================================================ FILE: benchmarks/flask-nodejs/source/.gitignore ================================================ ignore_me **/__pycache__/ ================================================ FILE: benchmarks/flask-nodejs/source/app.py ================================================ from flask import Flask, request, jsonify from bcrypt import hashpw, gensalt from db import read_items, write_items app = Flask(__name__) @app.route('/', methods=['GET']) def hello_world(): return "Hello World!" @app.route('/grocery_items', methods=['GET']) def get_grocery_items(): try: grocery_items = read_items() items = [{"id": item["id"], "name": item["name"], "price": item["price"]} for item in grocery_items] return jsonify(items) except Exception as e: return e, 500 @app.route('/grocery_items', methods=['POST']) def add_grocery_item(): try: new_item = request.json print(new_item["id"],new_item,flush=True) grocery_items = read_items() if new_item not in grocery_items: grocery_items.append(new_item) write_items(grocery_items) return "Successfully added item", 201 except Exception as e: return e, 500 @app.route('/grocery_items/', methods=['DELETE']) def delete_grocery_item(item_id): try: grocery_items = read_items() grocery_items = [item for item in grocery_items if item["id"] != item_id] write_items(grocery_items) return "Successfully deleted item", 200 except Exception as e: return e, 500 @app.route('/hashpassword/', methods=['GET']) def hash_password(password): try: return hashpw(password.encode('utf-8'), gensalt()).decode('utf-8') except Exception as e: return e, 500 if __name__ == '__main__': app.run(debug=True) ================================================ FILE: benchmarks/flask-nodejs/source/db.py ================================================ import json def read_items(): with open('storage/items.json') as f: grocery_items = json.load(f) return grocery_items def write_items(grocery_items): with open('storage/items.json', 'w') as f: json.dump(grocery_items, f) ================================================ FILE: benchmarks/flask-nodejs/source/requirements.txt ================================================ flask bcrypt ================================================ FILE: benchmarks/flask-nodejs/source/storage/items.json ================================================ [{"id": 1, "name": "Milk", "price": 2.5}, {"id": 2, "name": "Eggs", "price": 3.0}, {"id": 3, "name": "Bread", "price": 1.5}] ================================================ FILE: benchmarks/flask-rust/source/.gitignore ================================================ ignore_me **/__pycache__/ ================================================ FILE: benchmarks/flask-rust/source/app.py ================================================ from flask import Flask, request, jsonify from bcrypt import hashpw, gensalt from db import read_items, write_items app = Flask(__name__) @app.route('/', methods=['GET']) def hello_world(): return "Hello World!" @app.route('/grocery_items', methods=['GET']) def get_grocery_items(): try: grocery_items = read_items() items = [{"id": item["id"], "name": item["name"], "price": item["price"]} for item in grocery_items] return jsonify(items) except Exception as e: return e, 500 @app.route('/grocery_items', methods=['POST']) def add_grocery_item(): try: new_item = request.json print(new_item["id"],new_item,flush=True) grocery_items = read_items() if new_item not in grocery_items: grocery_items.append(new_item) write_items(grocery_items) return "Successfully added item", 201 except Exception as e: return e, 500 @app.route('/grocery_items/', methods=['DELETE']) def delete_grocery_item(item_id): try: grocery_items = read_items() grocery_items = [item for item in grocery_items if item["id"] != item_id] write_items(grocery_items) return "Successfully deleted item", 200 except Exception as e: return e, 500 @app.route('/hashpassword/', methods=['GET']) def hash_password(password): try: return hashpw(password.encode('utf-8'), gensalt()).decode('utf-8') except Exception as e: return e, 500 if __name__ == '__main__': app.run(debug=True) ================================================ FILE: benchmarks/flask-rust/source/db.py ================================================ import json def read_items(): with open('storage/items.json') as f: grocery_items = json.load(f) return grocery_items def write_items(grocery_items): with open('storage/items.json', 'w') as f: json.dump(grocery_items, f) ================================================ FILE: benchmarks/flask-rust/source/requirements.txt ================================================ flask bcrypt ================================================ FILE: benchmarks/flask-rust/source/storage/items.json ================================================ [{"id": 1, "name": "Milk", "price": 2.5}, {"id": 2, "name": "Eggs", "price": 3.0}, {"id": 3, "name": "Bread", "price": 1.5}] ================================================ FILE: gpt_migrate/ai.py ================================================ from langchain.chat_models import ChatOpenAI from config import OPENAI_API_KEY import os import openai from utils import parse_code_string from litellm import completion openai.api_key = os.getenv("OPENAI_API_KEY") class AI: def __init__(self, model='openrouter/openai/gpt-4-32k', temperature=0.1, max_tokens=10000): self.temperature = temperature self.max_tokens = max_tokens self.model_name = model try: _ = ChatOpenAI(model_name=model) # check to see if model is available to user except Exception as e: print(e) self.model_name = "gpt-3.5-turbo" def write_code(self, prompt): message=[{"role": "user", "content": str(prompt)}] response = completion( messages=message, stream=False, model=self.model_name, max_tokens=self.max_tokens, temperature=self.temperature ) if response["choices"][0]["message"]["content"].startswith("INSTRUCTIONS:"): return ("INSTRUCTIONS:","",response["choices"][0]["message"]["content"][14:]) else: code_triples = parse_code_string(response["choices"][0]["message"]["content"]) return code_triples def run(self, prompt): message=[{"role": "user", "content": str(prompt)}] response = completion( messages=message, stream=True, model=self.model_name, max_tokens=self.max_tokens, temperature=self.temperature ) chat = "" for chunk in response: delta = chunk["choices"][0]["delta"] msg = delta.get("content", "") chat += msg return chat ================================================ FILE: gpt_migrate/config.py ================================================ import os ''' Environment variables ''' OPENAI_API_KEY = os.environ.get("OPENAI_API_KEY") ''' Global variables ''' MAX_ERROR_MESSAGE_CHARACTERS = 5000 MAX_DOCKER_LOG_CHARACTERS = 2000 ''' Prompt directory ''' HIERARCHY = "HIERARCHY" GUIDELINES = "p1_guidelines/guidelines" WRITE_CODE = "p2_actions/write_code" CREATE_DOCKER = "p3_setup/create_target_docker" GET_EXTERNAL_DEPS = "p3_migrate/1_get_external_deps" GET_INTERNAL_DEPS = "p3_migrate/2_get_internal_deps" WRITE_MIGRATION = "p3_migrate/3_write_migration" ADD_DOCKER_REQUIREMENTS = "p3_migrate/4_add_docker_requirements" REFINE_DOCKERFILE = "p3_migrate/5_refine_target_docker" GET_FUNCTION_SIGNATURES = "p3_migrate/6_get_function_signatures" CREATE_TESTS = "p3_test/create_tests" DEBUG_DOCKERFILE = "p3_debug/debug_target_docker" IDENTIFY_ACTION = "p3_debug/identify_action" MOVE_FILES = "p3_debug/move_files" CREATE_FILE = "p3_debug/create_file" IDENTIFY_FILE = "p3_debug/identify_file" DEBUG_FILE = "p3_debug/debug_file" DEBUG_TESTFILE = "p3_debug/debug_testfile" HUMAN_INTERVENTION = "p3_debug/human_intervention" MULTIFILE = "p4_output_formats/multi_file" SINGLEFILE = "p4_output_formats/single_file" FILENAMES = "p4_output_formats/filenames" ''' Living list of types of files that should be excluded from being copied over ''' EXCLUDED_FILES = [ # Docker 'Dockerfile', # Python 'requirements.txt', '__pycache__/', # JS 'package.json', 'package-lock.json', 'yarn.lock', 'node_modules/' # Rust 'Cargo.toml' # TODO: add more ] ''' Living list of file extensions that should be copied over ''' INCLUDED_EXTENSIONS = ( '.env', '.txt', '.json', '.csv', '.rdb', '.db' # TODO: add more ) TREE_SITTER_REPO_STUB = "https://github.com/tree-sitter/tree-sitter-" EXTENSION_TO_TREE_SITTER_GRAMMAR_REPO = { 'py': TREE_SITTER_REPO_STUB + "python", 'js': TREE_SITTER_REPO_STUB + "javascript", 'java': TREE_SITTER_REPO_STUB + "java", 'rb': TREE_SITTER_REPO_STUB + "ruby", 'php': TREE_SITTER_REPO_STUB + "php", 'cs': TREE_SITTER_REPO_STUB + "c-sharp", 'go': TREE_SITTER_REPO_STUB + "go", 'rs': TREE_SITTER_REPO_STUB + "rust", 'cpp': TREE_SITTER_REPO_STUB + "cpp", 'cc': TREE_SITTER_REPO_STUB + "cpp", 'cxx': TREE_SITTER_REPO_STUB + "cpp", 'c': TREE_SITTER_REPO_STUB + "c", 'swift': TREE_SITTER_REPO_STUB + "swift", 'scala': TREE_SITTER_REPO_STUB + "scala", 'ts': TREE_SITTER_REPO_STUB + "typescript", 'tsx': TREE_SITTER_REPO_STUB + "typescript", 'jsx': TREE_SITTER_REPO_STUB + "javascript", 'hs': TREE_SITTER_REPO_STUB + "haskell", 'jl': TREE_SITTER_REPO_STUB + "julia", } EXTENSION_TO_LANGUAGE = { 'py': 'Python', 'js': 'JavaScript', 'java': 'Java', 'rb': 'Ruby', 'php': 'PHP', 'cs': 'C#', 'go': 'Go', 'rs': 'Rust', 'cpp': 'C++', 'cc': 'C++', 'cxx': 'C++', 'c': 'C', 'swift': 'Swift', 'm': 'Objective-C', 'kt': 'Kotlin', 'scala': 'Scala', 'pl': 'Perl', 'pm': 'Perl', 'r': 'R', 'lua': 'Lua', 'groovy': 'Groovy', 'ts': 'TypeScript', 'tsx': 'TypeScript', 'jsx': 'JavaScript', 'dart': 'Dart', 'elm': 'Elm', 'erl': 'Erlang', 'ex': 'Elixir', 'fs': 'F#', 'hs': 'Haskell', 'jl': 'Julia', 'nim': 'Nim', } ================================================ FILE: gpt_migrate/main.py ================================================ import typer from utils import detect_language, build_directory_structure import os import time as time from collections import defaultdict from ai import AI from steps.setup import create_environment from steps.migrate import get_dependencies, write_migration, add_env_files from steps.test import run_dockerfile, create_tests, validate_tests, run_test from steps.debug import debug_testfile, debug_error app = typer.Typer() class Globals: def __init__(self, sourcedir, targetdir, sourcelang, targetlang, sourceentry, source_directory_structure, operating_system, testfiles, sourceport, targetport, guidelines, ai): self.sourcedir = sourcedir self.targetdir = targetdir self.sourcelang = sourcelang self.targetlang = targetlang self.sourceentry = sourceentry self.source_directory_structure = source_directory_structure self.operating_system=operating_system self.testfiles = testfiles self.sourceport = sourceport self.targetport = targetport self.guidelines = guidelines self.ai = ai @app.command() def main( model: str = typer.Option('openrouter/openai/gpt-4-32k', help="Large Language Model to be used. Default is 'openrouter/openai/gpt-4-32k'. To use OpenAI directly with your API key, use 'gpt-4-32k'."), temperature: float = typer.Option(0, help="Temperature setting for the AI model."), sourcedir: str = typer.Option("../benchmarks/flask-nodejs/source", help="Source directory containing the code to be migrated."), sourcelang: str = typer.Option(None, help="Source language or framework of the code to be migrated."), sourceentry: str = typer.Option("app.py", help="Entrypoint filename relative to the source directory. For instance, this could be an app.py or main.py file for Python."), targetdir: str = typer.Option("../benchmarks/flask-nodejs/target", help="Directory where the migrated code will live."), targetlang: str = typer.Option("nodejs", help="Target language or framework for migration."), operating_system: str = typer.Option("linux", help="Operating system for the Dockerfile. Common options are 'linux' or 'windows'."), testfiles: str = typer.Option("app.py", help="Comma-separated list of files that have functions to be tested. For instance, this could be an app.py or main.py file for Python app where your REST endpoints are. Include the full relative path."), sourceport: int = typer.Option(None, help="(Optional) port for testing the unit tests file against the original app."), targetport: int = typer.Option(8080, help="Port for testing the unit tests file against the migrated app."), guidelines: str = typer.Option("", help="Stylistic or small functional guidelines that you'd like to be followed during the migration. For instance, \"Use tabs, not spaces\"."), step: str = typer.Option("all", help="Step to run. Options are 'setup', 'migrate', 'test', 'all'.") ): ai = AI( model=model, temperature=temperature, ) sourcedir = os.path.abspath(sourcedir) targetdir = os.path.abspath(targetdir) os.makedirs(targetdir, exist_ok=True) detected_language = detect_language(sourcedir) if not sourcelang else sourcelang if not sourcelang: if detected_language: is_correct = typer.confirm(f"Is your source project a {detected_language} project?") if is_correct: sourcelang = detected_language else: sourcelang = typer.prompt("Please enter the correct language for the source project") else: sourcelang = typer.prompt("Unable to detect the language of the source project. Please enter it manually") if not os.path.exists(os.path.join(sourcedir, sourceentry)): sourceentry = typer.prompt("Unable to find the entrypoint file. Please enter it manually. This must be a file relative to the source directory.") source_directory_structure = build_directory_structure(sourcedir) globals = Globals(sourcedir, targetdir, sourcelang, targetlang, sourceentry, source_directory_structure, operating_system, testfiles, sourceport, targetport, guidelines, ai) typer.echo(typer.style(f"◐ Reading {sourcelang} project from directory '{sourcedir}', with entrypoint '{sourceentry}'.", fg=typer.colors.BLUE)) time.sleep(0.3) typer.echo(typer.style(f"◑ Outputting {targetlang} project to directory '{targetdir}'.", fg=typer.colors.BLUE)) time.sleep(0.3) typer.echo(typer.style("Source directory structure: \n\n" + source_directory_structure, fg=typer.colors.BLUE)) ''' 1. Setup ''' if step in ['setup', 'all']: # Set up environment (Docker) create_environment(globals) ''' 2. Migration ''' if step in ['migrate', 'all']: target_deps_per_file = defaultdict(list) def migrate(sourcefile, globals, parent_file=None): # recursively work through each of the files in the source directory, starting with the entrypoint. internal_deps_list, external_deps_list = get_dependencies(sourcefile=sourcefile,globals=globals) for dependency in internal_deps_list: migrate(dependency, globals, parent_file=sourcefile) file_name = write_migration(sourcefile, external_deps_list, target_deps_per_file.get(sourcefile), globals) target_deps_per_file[parent_file].append(file_name) migrate(sourceentry, globals) add_env_files(globals) ''' 3. Testing ''' if step in ['test', 'all']: while True: result = run_dockerfile(globals) if result=="success": break debug_error(result,"",globals) for testfile in globals.testfiles.split(','): generated_testfile = create_tests(testfile,globals) if globals.sourceport: while True: result = validate_tests(generated_testfile, globals) time.sleep(0.3) if result=="success": break debug_testfile(result,testfile,globals) while True: result = run_test(generated_testfile, globals) if result=="success": break debug_error(result,globals.testfiles,globals) run_dockerfile(globals) time.sleep(1) # wait for docker to spin up typer.echo(typer.style("All tests complete. Ready to rumble. 💪", fg=typer.colors.GREEN)) if __name__ == "__main__": app() ================================================ FILE: gpt_migrate/memory/external_dependencies ================================================ rocket serde serde_json bcrypt rusqlite serde_json ================================================ FILE: gpt_migrate/parser.py ================================================ import subprocess import typer from yaspin import yaspin from pathlib import Path from tree_sitter import Language, Parser, Node from collections.abc import Iterator from config import EXTENSION_TO_TREE_SITTER_GRAMMAR_REPO, EXTENSION_TO_LANGUAGE def decompose_file(file_path: str) -> Iterator[Node]: # Do a first-level parse tree decomposition of the file at file_path with yaspin(text="Decomposing file", spinner="dots") as spinner: repo_url = EXTENSION_TO_TREE_SITTER_GRAMMAR_REPO.get(file_path.split('.')[-1]) if not repo_url: success_text = typer.style("Couldn't find tree-sitter grammar for programming language {}. Aborting decomposition of file.".format(EXTENSION_TO_LANGUAGE.get(file_path.split('.')[-1])), fg=typer.colors.RED) typer.echo(success_text) repo_name = repo_url.split('/')[-1] if not Path("cache/tree-sitter/" + repo_name).exists(): Path("cache/tree-sitter").mkdir(parents=True, exist_ok=True) result = subprocess.run(["git", "clone", repo_url], cwd="cache/tree-sitter", stdout=subprocess.PIPE, stderr=subprocess.STDOUT, check=True, text=True) grammar_lib_path = Path("cache/tree-sitter/my-languages.so") if grammar_lib_path.exists(): grammar_lib_path.unlink() Language.build_library( 'cache/tree-sitter/my-languages.so', ['cache/tree-sitter/' + repo_name] ) lang = Language('cache/tree-sitter/my-languages.so', repo_url.split('-')[-1]) parser = Parser() parser.set_language(lang) with open(file_path) as f: source_code = f.read() tree = parser.parse(bytes(source_code, "utf8")) root_node = tree.root_node for child in root_node.children: yield child spinner.ok("✅ ") ================================================ FILE: gpt_migrate/prompts/HIERARCHY ================================================ The following prompt is a composition of prompt sections, each with different preference levels. Higher preference levels override lower preference levels. The lowest preference level, PREFERENCE LEVEL 1, will likely be a broad guideline of some sort. Prompt sections with higher preference levels are likely to be more specific instructions that would override sections with lower preference levels. Each prompt section will start with "PREFERENCE LEVEL (level)". ================================================ FILE: gpt_migrate/prompts/p1_guidelines/guidelines ================================================ \n\n PREFERENCE LEVEL 1 Here are the guidelines for this prompt: 1. Follow the output instructions precisely and do not make any assumptions. Your output will not be read by a human; it will be directly input into a computer for literal processing. Adding anything else or deviating from the instructions will cause the output to fail. 2. Think through the answer to each prompt step by step to ensure that the output is perfect; there is no room for error. 3. Do not use any libraries, frameworks, or projects that are not well-known and well-documented, unless they are explicitly mentioned in the instructions or in the prompt. 4. In general, use comments in code only sparingly. ================================================ FILE: gpt_migrate/prompts/p2_actions/write_code ================================================ \n\n PREFERENCE LEVEL 2 You are a pragmatic principal engineer at Google. You are about to get instructions for code to write. This code must be as simple and easy to understand, while still fully expressing the functionality required. Please note that the code should be complete and fully functional. No placeholders. However, only write what you are asked to write. For instance, if you're asked to write a function, only write the function; DO NOT include import statements. We will do those separately. Please strictly follow this styling guideline with no deviations. Variables will always be snake_case; either capital or lowercase. Functions will always be camelCase. Classes will always be PascalCase. Please follow this guideline even if the source code does not follow it. Finally, please follow these guidelines: {guidelines} ================================================ FILE: gpt_migrate/prompts/p3_debug/create_file ================================================ \n\n PREFERENCE LEVEL 3 You are a principal software engineer at Google. The current app is having trouble running. Consider the below error and current directory structure. It has been determined that creating a new file in the directory may help. Please provide a file with the full relative path name in the format you're about to see. Finally, the script should only be one line. For multiple commands, please use the && operator. ``` Error message: ``` {error_message} ``` Current directory structure: ``` {target_directory_structure} ``` ================================================ FILE: gpt_migrate/prompts/p3_debug/debug_file ================================================ \n\n PREFERENCE LEVEL 3 You are a principal software engineer at Google. We are doing a migration to {targetlang} from {sourcelang}. The current Docker app is having trouble running. Consider the below error and the current version of {file_name}, which is at least partially responsible for this bug. Please rewrite this file to fix the bug. Error message: ``` {error_message} ``` Docker logs: ``` {docker_logs} ``` Current {file_name}: ``` {old_file_content} ``` Other files that may be relevant: {relevant_files} ================================================ FILE: gpt_migrate/prompts/p3_debug/debug_target_docker ================================================ \n\n PREFERENCE LEVEL 3 You are a principal software engineer at Google with particular expertise in Docker environments. The current Docker environment is having trouble running. Consider the below error, current Dockerfile, and directory structure. Your job is to create a comprehensive Dockerfile which will allow the app to run in a Docker environment. Error message: ``` {error_message} ``` Target directory structure: ``` {target_directory_structure} ``` Current Dockerfile: ``` {dockerfile_content} ``` ================================================ FILE: gpt_migrate/prompts/p3_debug/debug_testfile ================================================ \n\n PREFERENCE LEVEL 3 You are a principal software engineer at Google. We are writing {file_name} using unittest against the following source file. The test script is reporting the following error(s), see below. Consider the below error and the current version of {file_name}. Please rewrite this file to fix the bug. Try to ensure that for whatever tests you write, the databases involved end up at their original state after running all of the tests. Create one test function per endpoint or testable function. Error message: ``` {error_message} ``` Current {file_name}: ``` {old_file_content} ``` Source test file(s): {relevant_files} ================================================ FILE: gpt_migrate/prompts/p3_debug/human_intervention ================================================ \n\n PREFERENCE LEVEL 3 You are a principal software engineer at Google who is pair programming with a partner. You've been given the following error message: ``` {error_message} ``` Here are some relevant files: {relevant_files} And finally, here is the current directory structure: ``` {target_directory_structure} ``` Please respond in simple English with a description of the error and a suggested fix. ================================================ FILE: gpt_migrate/prompts/p3_debug/identify_action ================================================ \n\n PREFERENCE LEVEL 3 You are a principal software engineer at Google pair programming with a partner. The current Docker app is having trouble running. Consider the below error and the directory structure. Your job is to identify which action to take from the following: MOVE_FILES, CREATE_FILES, EDIT_FILES. Your output must be a comma-separated list of one or more of the options, for example: MOVE_FILES,EDIT_FILES. If copying, moving, or deleting a file or directory will help, output "MOVE_FILES". If creating one or more new files will help you resolve the issue, output "CREATE_FILES". If editing one or more existing files can help resolve the issue, output "EDIT_FILES". It's highly important that your output is a comma-separated list of MOVE_FILES, CREATE_FILES, and/or EDIT_FILES. Nothing else. Error message: ``` {error_message} ``` Directory structure: ``` {target_directory_structure} ``` ================================================ FILE: gpt_migrate/prompts/p3_debug/identify_file ================================================ \n\n PREFERENCE LEVEL 3 You are a principal software engineer at Google. The current Docker app is having trouble running. Consider the below error and the directory structure. Your job is to identify which file is responsible for this error. If multiple files, please list them in comma-separated values and include the whole relative path, for example: file1.ext,directory1/file2.ext,file3.ext... Do not select any files in the gpt_migrate directory, unless the gpt_migrate directory is specifically mentioned below in the error message (in this case, include gpt_migrate/filename). If the error is from one of these, you'll need to fix the correspondingly named file in the main directory. Error message: ``` {error_message} ``` Docker logs: ``` {docker_logs} ``` Directory structure: ``` {target_directory_structure} ``` The output format should be either a single filename or a comma-separated list of filenames, and nothing else. If there are none that are obvious, please write only "NONE FOUND" and nothing else. ================================================ FILE: gpt_migrate/prompts/p3_debug/move_files ================================================ \n\n PREFERENCE LEVEL 3 You are a principal software engineer at Google. The current app is having trouble running. Consider the below error and current directory structure. It has been determined that moving (or renaming), copying, or removing one or more files from the directory may help. Please provide a complete shell script that will do so using only the commands `cp`, `mv`, or `rm`. We are not able to change directories so please use the full path in all commands and do not use `cd`. This should be written for the {operating_system} operating system. This will be run directly in the terminal. Finally, the script should only be one line. For multiple commands, please use the && operator. ``` Error message: ``` {error_message} ``` Current directory structure: ``` {target_directory_structure} ``` Full path: ``` {current_full_path} ``` ================================================ FILE: gpt_migrate/prompts/p3_migrate/1_get_external_deps ================================================ \n\n PREFERENCE LEVEL 3 You are a principal software engineer at Google with particular expertise migrating codebases from {sourcelang} to {targetlang}. We are doing a migration from {sourcelang} to {targetlang}. Please respond only with a comma-separated list of {targetlang} libraries you would want to have installed in a {targetlang} project based on the following {sourcelang} file: ``` {sourcefile_content} ``` If there are no outside libraries, answer only NONE. If there are libraries, please list them in the following format: dep1,dep2,dep3... Please do not include any other information in your answer. The content of your output will be directly read into a file and any deviation will cause this process to fail. ================================================ FILE: gpt_migrate/prompts/p3_migrate/2_get_internal_deps ================================================ \n\n PREFERENCE LEVEL 3 You are a principal software engineer at Google with particular expertise migrating codebases from {sourcelang} to {targetlang}. We are doing a migration from {sourcelang} to {targetlang}. Please respond only with a comma-separated list of any internal dependency files. For instance, if we were in python, `from db import mongo` would probably indicate there is an internal dependency file called db.py. Some may be external; ignore those. For instance, if we were in python, `import pandas` would be an external dependency and it should be ignored. We are currently in {sourcefile}. Directory structure: ``` {source_directory_structure} ``` {sourcefile}: ``` {sourcefile_content} ``` If there are no internal dependency files, answer only NONE. If there are internal dependency files, please list their paths relative to the root of this directory in the following comma-separated format as such: dep1.ext,folder1/dep2.ext,dep3.ext... Please do not include any other information in your answer. The content of your output will be directly read into a file and any deviation will cause this process to fail. ================================================ FILE: gpt_migrate/prompts/p3_migrate/3_write_migration ================================================ \n\n PREFERENCE LEVEL 3 You are a principal software engineer at Google with particular expertise migrating codebases from {sourcelang} to {targetlang}. We are doing a migration from {sourcelang} to {targetlang}. You are allowed to use the following external libraries, but no other external libraries: {external_deps}. You will be given the current target directory structure of the {targetlang} project, the source directory structure of the existing {sourcelang} project, and the contents of the {sourcelang} file. Please use the below code format and name the file, variables, functions, etc. to be consistent with the existing {sourcelang} file where possible. The only exception is if this is an entrypoint file and {targetlang} requires a certain naming convention, such as main.ext etc. For the filename, include the full relative path if applicable. If the {sourcelang} code imports internal libraries from a given location, take special care to preserve this topology in the code you write for the {targetlang} project and use the functions in the internal libraries accordingly. Any port listening should be on 8080. Please ensure that all functions and variables are available to other files that may call them. Current target directory structure, which is under active development and may have files added later which you can import from: ``` {target_directory_structure} ``` Existing source directory structure: ``` {source_directory_structure} ``` Existing {sourcelang} file, {sourcefile}: ``` {sourcefile_content} ``` Available internal functions in {targetlang}, their signatures and descriptions: ``` {targetlang_function_signatures} ``` ================================================ FILE: gpt_migrate/prompts/p3_migrate/4_add_docker_requirements ================================================ \n\n PREFERENCE LEVEL 3 You are a principal software engineer at Google with particular expertise in Docker environments. Consider the below Dockerfile and create a file for external dependencies. Some languages require a specific directory and/or file name, and to address this in the file for external dependencies. The current directory structure is as follows: ``` {target_directory_structure} ``` If this will be an issue for {targetlang}, please handle for this in the external dependencies file. Dockerfile: ``` {dockerfile_content} ``` External dependencies: ``` {external_deps} ``` ================================================ FILE: gpt_migrate/prompts/p3_migrate/5_refine_target_docker ================================================ \n\n PREFERENCE LEVEL 3 You are a principal software engineer at Google with particular expertise in Docker environments. Consider the below Dockerfile, dependencies file, and directory structure. Your job is to create a comprehensive Dockerfile which will allow the app to run in a Docker environment the first time. Target directory structure: ``` {target_directory_structure} ``` Current Dockerfile: ``` {dockerfile_content} ``` External dependencies, {external_deps_name}: ``` {external_deps_content} ``` ================================================ FILE: gpt_migrate/prompts/p3_migrate/6_get_function_signatures ================================================ \n\n PREFERENCE LEVEL 3 You are a principal software engineer at Google with particular expertise migrating codebases from {sourcelang} to {targetlang}. We are doing a migration from {sourcelang} to {targetlang}. As an intermediate step, we have to extract all function signatures from a {targetlang} file. Make sure to include types as well as default arguments. Futhermore, we have to give a concise description of the function. It should be clear how to call the function from the description and signature only. Here is the {targetlang} file: ``` {targetfile_content} ``` If there are no functions, answer only NONE. If there are functions, please respond in JSON format. Here is an example for the structure of the response for a hypothetical Python file. [ {{ "signature": "is_prime(number: int) -> bool", "description": "Determines if the number is a prime number", }}, {{ "signature": "get_current_weather(location: string, unit: string = "celsius") -> int", "description": "Get the current weather in a given location. The unit can be optionally specified and should be either celsius or fahrenheit.", }} ] Please do not include any other information in your answer. The content of your output will be directly read into a file and any deviation will cause this process to fail. ================================================ FILE: gpt_migrate/prompts/p3_setup/create_target_docker ================================================ \n\n PREFERENCE LEVEL 3 Please create a Dockerfile for a generic app in the following framework: {targetlang}. This new app is being transpiled to {targetlang} from {sourcelang}, where the entrypoint file name is {sourceentry}. Please use the same file name besides the extension if in a different language, unless {targetlang} requires a certain naming convention for the entrypoint file, such as main.ext etc. Be sure to include a dependencies installation step with your choice of file name. No need to write any comments. Exposed port should be 8080. ================================================ FILE: gpt_migrate/prompts/p3_test/create_tests ================================================ \n\n PREFERENCE LEVEL 3 You are a principal software engineer at Google. You're responsible for creating a set of tests for this piece of code using Python. The tests should cover each main function - execute an input, check the output, repeat for each function. Assume this is exposed on port {targetport} if applicable. Please write a set of tests that can be executed either as a python file or as a shell script that will validate or invalidate this file. Please ensure that the error messages for the test file are clear and descriptive. Use unittest. Try to ensure that for whatever tests you write, the databases involved end up at their original state after running all of the tests. Create one test function per endpoint or testable function. Finally, in the logs, if the test fails, please have it log or print the response (instead of just asserting something). This will help us debug the issue. ``` {old_file_content} ``` ================================================ FILE: gpt_migrate/prompts/p4_output_formats/file_debug ================================================ \n\n PREFERENCE LEVEL 4 We will be using the output you provide as-is to create new files OR provide natural language instructions for how to debug the problem. If your output is a new file, please be precise and do not include any other text. Your output needs to be ONE file; if your output contains multiple files, it will break the system. Your output should consist ONLY of the file name, language, and code, in the following format: file_name.ext ```language CODE ``` If you decide that the problem cannot be fixed by simply changing a file, please provide natural language instructions for how to debug the problem. The format should be "INSTRUCTIONS: " followed by the text. For example: INSTRUCTIONS: You need to change the file name from "file_name.ext" to "new_file_name.ext" and then run the following command: "python3 file_name.ext". ================================================ FILE: gpt_migrate/prompts/p4_output_formats/filenames ================================================ \n\n PREFERENCE LEVEL 4 Please respond only with a comma-separated list of items: file.ext,dir1/file2.ext,file3.ext... Please do not include any other information in your answer. The content of your output will be directly read into a file and any deviation will cause this process to fail. If your list is empty, please instead write only NONE. ================================================ FILE: gpt_migrate/prompts/p4_output_formats/multi_file ================================================ \n\n PREFERENCE LEVEL 4 We will be using the output you provide as-is to create new files, so please be precise and do not include any other text. Your output should consist ONLY of the filename, language, and code, in the following format, separated by three dashes (---) for multiple files as shown: Filename.ext ```language CODE ``` --- Filename.ext ```language CODE ``` --- Filename.ext ```language CODE ``` --- ... ================================================ FILE: gpt_migrate/prompts/p4_output_formats/single_file ================================================ \n\n PREFERENCE LEVEL 4 We will be using the output you provide as-is to create new files, so please be precise and do not include any other text. Your output needs to be ONE file; if your output contains multiple files, it will break the system. Your output should consist ONLY of the file name, language, and code, in the following format: file_name.ext ```language CODE ``` ================================================ FILE: gpt_migrate/requirements.txt ================================================ typer langchain yaspin openai tree_sitter litellm==0.1.213 pydantic==1.10.8 ================================================ FILE: gpt_migrate/steps/__init__.py ================================================ ================================================ FILE: gpt_migrate/steps/debug.py ================================================ from utils import prompt_constructor, llm_write_file, llm_run, build_directory_structure, construct_relevant_files from config import HIERARCHY, GUIDELINES, WRITE_CODE, IDENTIFY_ACTION, MOVE_FILES, CREATE_FILE, IDENTIFY_FILE, DEBUG_FILE, DEBUG_TESTFILE, HUMAN_INTERVENTION, SINGLEFILE, FILENAMES, MAX_ERROR_MESSAGE_CHARACTERS, MAX_DOCKER_LOG_CHARACTERS import os import typer import subprocess def debug_error(error_message,relevant_files,globals): identify_action_template = prompt_constructor(HIERARCHY, GUIDELINES, IDENTIFY_ACTION) prompt = identify_action_template.format(error_message=error_message[-min(MAX_ERROR_MESSAGE_CHARACTERS, len(error_message)):], target_directory_structure=build_directory_structure(globals.targetdir)) actions = llm_run(prompt, waiting_message=f"Planning actions for debugging...", success_message="", globals=globals) action_list = actions.split(',') if "MOVE_FILES" in action_list: if not os.path.exists(os.path.join(globals.targetdir, 'gpt_migrate')): os.makedirs(os.path.join(globals.targetdir, 'gpt_migrate')) move_files_template = prompt_constructor(HIERARCHY, GUIDELINES, WRITE_CODE, MOVE_FILES, SINGLEFILE) prompt = move_files_template.format(error_message=error_message[-min(MAX_ERROR_MESSAGE_CHARACTERS, len(error_message)):], target_directory_structure=build_directory_structure(globals.targetdir), current_full_path=globals.targetdir, operating_system=globals.operating_system, guidelines=globals.guidelines) file_name, language, shell_script_content = llm_write_file(prompt, target_path="gpt_migrate/debug.sh", waiting_message=f"Writing shell script...", success_message="Wrote debug.sh based on error message.", globals=globals) # execute shell script from file_content using subprocess. Check with user using shell commands before executing. if typer.confirm("GPT-Migrate wants to run this shell script to debug your application, which is also stored in gpt_migrate/debug.sh: \n\n"+shell_script_content+"\n\nWould you like to run it?"): try: result = subprocess.run(["bash", "gpt_migrate/debug.sh"], stdout=subprocess.PIPE, stderr=subprocess.STDOUT, check=True, text=True) print(result.stdout) except subprocess.CalledProcessError as e: print("ERROR: ",e.output) error_message = e.output error_text = typer.style("Something isn't right with your shell script. Please ensure it is valid and try again.", fg=typer.colors.RED) typer.echo(error_text) raise typer.Exit() if "EDIT_FILES" in action_list: if relevant_files != "": fileslist = globals.testfiles.split(',') files_to_construct = [] for file_name in fileslist: with open(os.path.join(globals.sourcedir, file_name), 'r') as file: file_content = file.read() files_to_construct.append(("migration_source/"+file_name, file_content)) relevant_files = construct_relevant_files(files_to_construct) identify_file_template = prompt_constructor(HIERARCHY, GUIDELINES, IDENTIFY_FILE, FILENAMES) docker_logs = subprocess.run(["docker", "logs", "gpt-migrate"], stdout=subprocess.PIPE, stderr=subprocess.STDOUT, check=True, text=True).stdout prompt = identify_file_template.format(error_message=error_message[-min(MAX_ERROR_MESSAGE_CHARACTERS, len(error_message)):], target_directory_structure=build_directory_structure(globals.targetdir), docker_logs=docker_logs[-min(MAX_DOCKER_LOG_CHARACTERS, len(docker_logs)):]), file_names = llm_run(prompt, waiting_message=f"Identifying files to debug...", success_message="", globals=globals) file_name_list = file_names.split(',') for file_name in file_name_list: old_file_content = "" try: with open(os.path.join(globals.targetdir, file_name), 'r') as file: old_file_content = file.read() except: print("File not found: "+file_name+". Please ensure the file exists and try again. You can resume the debugging process with the `--step test` flag.") raise typer.Exit() debug_file_template = prompt_constructor(HIERARCHY, GUIDELINES, WRITE_CODE, DEBUG_FILE, SINGLEFILE) prompt = debug_file_template.format(error_message=error_message[-min(MAX_ERROR_MESSAGE_CHARACTERS, len(error_message)):], file_name=file_name, old_file_content=old_file_content, targetlang=globals.targetlang, sourcelang=globals.sourcelang, docker_logs=docker_logs[-min(MAX_DOCKER_LOG_CHARACTERS, len(docker_logs)):], relevant_files=relevant_files, guidelines=globals.guidelines), _, language, file_content = llm_write_file(prompt, target_path=file_name, waiting_message=f"Debugging {file_name}...", success_message=f"Re-wrote {file_name} based on error message.", globals=globals) new_file_content = "" with open(os.path.join(globals.targetdir, file_name), 'r') as file: new_file_content = file.read() if new_file_content==old_file_content: require_human_intervention(error_message,construct_relevant_files([(file_name, new_file_content)]),globals) if "CREATE_FILE" in action_list: create_file_template = prompt_constructor(HIERARCHY, GUIDELINES, WRITE_CODE, CREATE_FILE, SINGLEFILE) prompt = create_file_template.format(error_message=error_message[-min(MAX_ERROR_MESSAGE_CHARACTERS, len(error_message)):], target_directory_structure=build_directory_structure(globals.targetdir), guidelines=globals.guidelines) new_file_name, language, file_content = llm_write_file(prompt, waiting_message=f"Creating a new file...", success_message="", globals=globals) success_text = typer.style(f"Created new file {new_file_name}.", fg=typer.colors.GREEN) typer.echo(success_text) def debug_testfile(error_message,testfile,globals): source_file_content = "" with open(os.path.join(globals.sourcedir, testfile), 'r') as file: source_file_content = file.read() relevant_files = construct_relevant_files([("migration_source/"+testfile, source_file_content)]) file_name = f"gpt_migrate/{testfile}.tests.py" try: with open(os.path.join(globals.targetdir, file_name), 'r') as file: old_file_content = file.read() except: print("File not found: "+file_name+". Please ensure the file exists and try again. You can resume the debugging process with the `--step test` flag.") raise typer.Exit() debug_file_template = prompt_constructor(HIERARCHY, GUIDELINES, WRITE_CODE, DEBUG_TESTFILE, SINGLEFILE) prompt = debug_file_template.format(error_message=error_message[-min(MAX_ERROR_MESSAGE_CHARACTERS, len(error_message)):], file_name=file_name, old_file_content=old_file_content, relevant_files=relevant_files, guidelines=globals.guidelines), _, language, file_content = llm_write_file(prompt, target_path=file_name, waiting_message=f"Debugging {file_name}...", success_message=f"Re-wrote {file_name} based on error message.", globals=globals) with open(os.path.join(globals.targetdir, file_name), 'r') as file: new_file_content = file.read() if new_file_content==old_file_content: require_human_intervention(error_message,construct_relevant_files([(file_name, new_file_content)]),globals) def require_human_intervention(error_message,relevant_files,globals): human_intervention_template = prompt_constructor(HIERARCHY, GUIDELINES, WRITE_CODE, HUMAN_INTERVENTION, SINGLEFILE) prompt = human_intervention_template.format(error_message=error_message[-min(MAX_ERROR_MESSAGE_CHARACTERS, len(error_message)):], relevant_files=relevant_files, target_directory_structure=build_directory_structure(globals.targetdir), guidelines=globals.guidelines) instructions = llm_run(prompt, waiting_message=f"Writing instructions for how to proceed...", success_message="", globals=globals) typer.echo(typer.style(f"GPT-Migrate is having some trouble debugging your app and requires human intervention. Below are instructions for how to fix your application.", fg=typer.colors.BLUE)) print(instructions) typer.echo(typer.style(f"Once the fix is implemented, you can pick up from the testing phase using the `--step test` flag.", fg=typer.colors.BLUE)) raise typer.Exit() ================================================ FILE: gpt_migrate/steps/migrate.py ================================================ from utils import prompt_constructor, llm_write_file, llm_run, build_directory_structure, copy_files, write_to_memory, read_from_memory, file_exists_in_memory, convert_sigs_to_string from config import HIERARCHY, GUIDELINES, WRITE_CODE, GET_EXTERNAL_DEPS, GET_INTERNAL_DEPS, ADD_DOCKER_REQUIREMENTS, REFINE_DOCKERFILE, WRITE_MIGRATION, SINGLEFILE, EXCLUDED_FILES, GET_FUNCTION_SIGNATURES from typing import List import os import json import typer def get_function_signatures(targetfiles: List[str], globals): ''' Get the function signatures and a one-sentence summary for each function ''' all_sigs = [] for targetfile in targetfiles: sigs_file_name = targetfile + "_sigs.json" if file_exists_in_memory(sigs_file_name): with open(os.path.join("memory", sigs_file_name), 'r') as f: sigs = json.load(f) all_sigs.extend(sigs) else: function_signatures_template = prompt_constructor(HIERARCHY, GUIDELINES, GET_FUNCTION_SIGNATURES) targetfile_content = "" with open(os.path.join(globals.targetdir, targetfile), 'r') as file: targetfile_content = file.read() prompt = function_signatures_template.format(targetlang=globals.targetlang, sourcelang=globals.sourcelang, targetfile_content=targetfile_content) sigs = json.loads(llm_run(prompt, waiting_message=f"Parsing function signatures for {targetfile}...", success_message=None, globals=globals)) all_sigs.extend(sigs) with open(os.path.join('memory', sigs_file_name), 'w') as f: json.dump(sigs, f) return all_sigs def get_dependencies(sourcefile, globals): ''' Get external and internal dependencies of source file ''' external_deps_prompt_template = prompt_constructor(HIERARCHY, GUIDELINES, GET_EXTERNAL_DEPS) internal_deps_prompt_template = prompt_constructor(HIERARCHY, GUIDELINES, GET_INTERNAL_DEPS) sourcefile_content = "" with open(os.path.join(globals.sourcedir, sourcefile), 'r') as file: sourcefile_content = file.read() prompt = external_deps_prompt_template.format(targetlang=globals.targetlang, sourcelang=globals.sourcelang, sourcefile_content=sourcefile_content) external_dependencies = llm_run(prompt, waiting_message=f"Identifying external dependencies for {sourcefile}...", success_message=None, globals=globals) external_deps_list = external_dependencies.split(',') if external_dependencies != "NONE" else [] write_to_memory("external_dependencies",external_deps_list) prompt = internal_deps_prompt_template.format(targetlang=globals.targetlang, sourcelang=globals.sourcelang, sourcefile=sourcefile, sourcefile_content=sourcefile_content, source_directory_structure=globals.source_directory_structure) internal_dependencies = llm_run(prompt, waiting_message=f"Identifying internal dependencies for {sourcefile}...", success_message=None, globals=globals) # Sanity checking internal dependencies to avoid infinite loops if sourcefile in internal_dependencies: typer.echo(typer.style(f"Warning: {sourcefile} seems to depend on itself. Automatically removing {sourcefile} from the list of internal dependencies.", fg=typer.colors.YELLOW)) internal_dependencies = internal_dependencies.replace(sourcefile, "") internal_deps_list = [dep for dep in internal_dependencies.split(',') if dep] if internal_dependencies != "NONE" else [] return internal_deps_list, external_deps_list def write_migration(sourcefile, external_deps_list, deps_per_file, globals) -> str: ''' Write migration file ''' sigs = get_function_signatures(deps_per_file, globals) if deps_per_file else [] write_migration_template = prompt_constructor(HIERARCHY, GUIDELINES, WRITE_CODE, WRITE_MIGRATION, SINGLEFILE) sourcefile_content = "" with open(os.path.join(globals.sourcedir, sourcefile), 'r') as file: sourcefile_content = file.read() prompt = write_migration_template.format(targetlang=globals.targetlang, targetlang_function_signatures=convert_sigs_to_string(sigs), sourcelang=globals.sourcelang, sourcefile=sourcefile, sourcefile_content=sourcefile_content, external_deps=','.join(external_deps_list), source_directory_structure=globals.source_directory_structure, target_directory_structure=build_directory_structure(globals.targetdir), guidelines=globals.guidelines) return llm_write_file(prompt, target_path=None, waiting_message=f"Creating migration file for {sourcefile}...", success_message=None, globals=globals)[0] def add_env_files(globals): ''' Copy all files recursively with included extensions from the source directory to the target directory in the same relative structure ''' copy_files(globals.sourcedir, globals.targetdir, excluded_files=EXCLUDED_FILES) ''' Add files required from the Dockerfile ''' add_docker_requirements_template = prompt_constructor(HIERARCHY, GUIDELINES, WRITE_CODE, ADD_DOCKER_REQUIREMENTS, SINGLEFILE) dockerfile_content = "" with open(os.path.join(globals.targetdir, 'Dockerfile'), 'r') as file: dockerfile_content = file.read() external_deps = read_from_memory("external_dependencies") prompt = add_docker_requirements_template.format(dockerfile_content=dockerfile_content, external_deps=external_deps, target_directory_structure=build_directory_structure(globals.targetdir), targetlang=globals.targetlang, guidelines=globals.guidelines) external_deps_name, _, external_deps_content = llm_write_file(prompt, target_path=None, waiting_message=f"Creating dependencies file required for the Docker environment...", success_message=None, globals=globals) ''' Refine Dockerfile ''' refine_dockerfile_template = prompt_constructor(HIERARCHY, GUIDELINES, WRITE_CODE, REFINE_DOCKERFILE, SINGLEFILE) prompt = refine_dockerfile_template.format(dockerfile_content=dockerfile_content, target_directory_structure=build_directory_structure(globals.targetdir), external_deps_name=external_deps_name, external_deps_content=external_deps_content, guidelines=globals.guidelines) llm_write_file(prompt, target_path="Dockerfile", waiting_message=f"Refining Dockerfile based on dependencies required for the Docker environment...", success_message="Refined Dockerfile with dependencies required for the Docker environment.", globals=globals) ================================================ FILE: gpt_migrate/steps/setup.py ================================================ from utils import prompt_constructor, llm_write_file from config import HIERARCHY, GUIDELINES, WRITE_CODE, CREATE_DOCKER, SINGLEFILE def create_environment(globals): ''' Create Dockerfile ''' docker_prompt_template = prompt_constructor(HIERARCHY, GUIDELINES, WRITE_CODE, CREATE_DOCKER, SINGLEFILE) prompt = docker_prompt_template.format(targetlang=globals.targetlang, sourcelang=globals.sourcelang, sourceentry=globals.sourceentry, guidelines=globals.guidelines) llm_write_file(prompt, target_path="Dockerfile", waiting_message="Creating your environment...", success_message=f"Created Docker environment for {globals.targetlang} project in directory '{globals.targetdir}'.", globals=globals) with open('memory/external_dependencies', 'w') as file: file.write('') ================================================ FILE: gpt_migrate/steps/test.py ================================================ from utils import prompt_constructor, llm_write_file, construct_relevant_files, find_and_replace_file from config import HIERARCHY, GUIDELINES, WRITE_CODE, CREATE_TESTS, SINGLEFILE import subprocess import typer import os import time as time from yaspin import yaspin from steps.debug import require_human_intervention def run_dockerfile(globals): try: with yaspin(text="Spinning up Docker container...", spinner="dots") as spinner: result = subprocess.run(["docker", "build", "-t", "gpt-migrate", globals.targetdir], stdout=subprocess.PIPE, stderr=subprocess.STDOUT, check=True, text=True) subprocess.run(["docker", "rm", "-f", "gpt-migrate"]) process = subprocess.Popen(["docker", "run", "-d", "-p", "8080:8080", "--name", "gpt-migrate", "gpt-migrate"], stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True) spinner.ok("✅ ") success_text = typer.style("Your Docker image is now running. GPT-Migrate will now start testing, and you can independently test as well. The application is exposed on port 8080.", fg=typer.colors.GREEN) typer.echo(success_text) return "success" except subprocess.CalledProcessError as e: print("ERROR: ",e.output) error_message = e.output error_text = typer.style("Something isn't right with Docker. Please ensure Docker is running and take a look over the Dockerfile; there may be errors. Once these are resolved, you can resume your progress with the `--step test` flag.", fg=typer.colors.RED) typer.echo(error_text) # have typer ask if the user would like to use AI to fix it? If so, call function fix(). if not, raise typer.Exit() if typer.confirm("Would you like GPT-Migrate to try to fix this?"): return error_message else: dockerfile_content = "" with open(os.path.join(globals.targetdir, 'Dockerfile'), 'r') as file: dockerfile_content = file.read() require_human_intervention(error_message,relevant_files=construct_relevant_files([("Dockerfile", dockerfile_content)]),globals=globals) raise typer.Exit() def create_tests(testfile,globals): # Makedir gpt_migrate in targetdir if it doesn't exist if not os.path.exists(os.path.join(globals.targetdir, 'gpt_migrate')): os.makedirs(os.path.join(globals.targetdir, 'gpt_migrate')) old_file_content = "" with open(os.path.join(globals.sourcedir, testfile), 'r') as file: old_file_content = file.read() create_tests_template = prompt_constructor(HIERARCHY, GUIDELINES, WRITE_CODE, CREATE_TESTS, SINGLEFILE) prompt = create_tests_template.format(targetport=globals.targetport, old_file_content=old_file_content, guidelines=globals.guidelines) _, _, file_content = llm_write_file(prompt, target_path=f"gpt_migrate/{testfile}.tests.py", waiting_message="Creating tests file...", success_message=f"Created {testfile}.tests.py file in directory gpt_migrate.", globals=globals) return f"{testfile}.tests.py" def validate_tests(testfile,globals): try: with yaspin(text="Validating tests...", spinner="dots") as spinner: # find all instances of globals.targetport in the testfile and replace with the port number globals.sourceport find_and_replace_file(os.path.join(globals.targetdir, f"gpt_migrate/{testfile}"), str(globals.targetport), str(globals.sourceport)) time.sleep(0.3) result = subprocess.run(["python3", os.path.join(globals.targetdir,f"gpt_migrate/{testfile}")], stdout=subprocess.PIPE, stderr=subprocess.STDOUT, check=True, text=True, timeout=15) spinner.ok("✅ ") print(result.stdout) find_and_replace_file(os.path.join(globals.targetdir, f"gpt_migrate/{testfile}"), str(globals.sourceport), str(globals.targetport)) typer.echo(typer.style(f"Tests validated successfully on your source app.", fg=typer.colors.GREEN)) return "success" except subprocess.CalledProcessError as e: find_and_replace_file(os.path.join(globals.targetdir, f"gpt_migrate/{testfile}"), str(globals.sourceport), str(globals.targetport)) print("ERROR: ",e.output) error_message = e.output error_text = typer.style(f"Validating {testfile} against your existing service failed. Please take a look at the error message and try to resolve the issue. Once these are resolved, you can resume your progress with the `--step test` flag.", fg=typer.colors.RED) typer.echo(error_text) if typer.confirm("Would you like GPT-Migrate to try to fix this?"): return error_message else: tests_content = "" with open(os.path.join(globals.targetdir, f"gpt_migrate/{testfile}"), 'r') as file: tests_content = file.read() require_human_intervention(error_message,relevant_files=construct_relevant_files([(f"gpt_migrate/{testfile}", tests_content)]),globals=globals) raise typer.Exit() except subprocess.TimeoutExpired as e: print(f"gpt_migrate/{testfile} timed out due to an unknown error and requires debugging.") return f"gpt_migrate/{testfile} timed out due to an unknown error and requires debugging." def run_test(testfile,globals): try: with yaspin(text="Running tests...", spinner="dots") as spinner: time.sleep(0.3) result = subprocess.run(["python3", os.path.join(globals.targetdir,f"gpt_migrate/{testfile}")], stdout=subprocess.PIPE, stderr=subprocess.STDOUT, check=True, text=True, timeout=15) spinner.ok("✅ ") print(result.stdout) success_text = typer.style(f"Tests passed for {testfile}!", fg=typer.colors.GREEN) typer.echo(success_text) return "success" except subprocess.CalledProcessError as e: print("ERROR: ",e.output) error_message = e.output error_text = typer.style(f"One or more tests in {testfile} failed. Please take a look at the error message and try to resolve the issue. Once these are resolved, you can resume your progress with the `--step test` flag.", fg=typer.colors.RED) typer.echo(error_text) if typer.confirm("Would you like GPT-Migrate to try to fix this?"): return error_message else: tests_content = "" with open(os.path.join(globals.targetdir, f"gpt_migrate/{testfile}"), 'r') as file: tests_content = file.read() require_human_intervention(error_message,relevant_files=construct_relevant_files([(f"gpt_migrate/{testfile}", tests_content)]),globals=globals) raise typer.Exit() except subprocess.TimeoutExpired as e: print(f"gpt_migrate/{testfile} timed out due to an unknown error and requires debugging.") return f"gpt_migrate/{testfile} timed out due to an unknown error and requires debugging." ================================================ FILE: gpt_migrate/utils.py ================================================ import os import typer from yaspin import yaspin from pathlib import Path from collections import Counter import fnmatch import re import shutil from config import INCLUDED_EXTENSIONS, EXTENSION_TO_LANGUAGE def detect_language(source_directory): file_extensions = [] for filenames in os.walk(source_directory): for filename in filenames[2]: ext = filename.split('.')[-1] file_extensions.append(ext) extension_counts = Counter(file_extensions) most_common_extension, _ = extension_counts.most_common(1)[0] # Determine the language based on the most common file extension language = EXTENSION_TO_LANGUAGE.get(most_common_extension, None) return language def prompt_constructor(*args): prompt = "" for arg in args: with open(os.path.abspath(f'prompts/{arg}'), 'r') as file: prompt += file.read().strip() return prompt def llm_run(prompt,waiting_message,success_message,globals): output = "" with yaspin(text=waiting_message, spinner="dots") as spinner: output = globals.ai.run(prompt) spinner.ok("✅ ") if success_message: success_text = typer.style(success_message, fg=typer.colors.GREEN) typer.echo(success_text) return output def llm_write_file(prompt,target_path,waiting_message,success_message,globals): file_content = "" with yaspin(text=waiting_message, spinner="dots") as spinner: file_name,language,file_content = globals.ai.write_code(prompt)[0] spinner.ok("✅ ") if file_name=="INSTRUCTIONS:": return "INSTRUCTIONS:","",file_content if target_path: with open(os.path.join(globals.targetdir, target_path), 'w') as file: file.write(file_content) else: with open(os.path.join(globals.targetdir, file_name), 'w') as file: file.write(file_content) if success_message: success_text = typer.style(success_message, fg=typer.colors.GREEN) typer.echo(success_text) else: success_text = typer.style(f"Created {file_name} at {globals.targetdir}", fg=typer.colors.GREEN) typer.echo(success_text) return file_name, language, file_content def llm_write_files(prompt,target_path,waiting_message,success_message,globals): file_content = "" with yaspin(text=waiting_message, spinner="dots") as spinner: results = globals.ai.write_code(prompt) spinner.ok("✅ ") for result in results: file_name,language,file_content = result if target_path: with open(os.path.join(globals.targetdir, target_path), 'w') as file: file.write(file_content) else: with open(os.path.join(globals.targetdir, file_name), 'w') as file: file.write(file_content) if not success_message: success_text = typer.style(f"Created {file_name} at {globals.targetdir}", fg=typer.colors.GREEN) typer.echo(success_text) if success_message: success_text = typer.style(success_message, fg=typer.colors.GREEN) typer.echo(success_text) return results def load_templates_from_directory(directory_path): templates = {} for filename in os.listdir(directory_path): with open(os.path.join(directory_path, filename), 'r') as file: key = os.path.splitext(filename)[0] templates[key] = file.read() return templates def parse_code_string(code_string): sections = code_string.split('---') pattern = re.compile(r'^(.+)\n```(.+?)\n(.*?)\n```', re.DOTALL) code_triples = [] for section in sections: match = pattern.match(section) if match: filename, language, code = match.groups() code_triples.append((section.split("\n```")[0], language.strip(), code.strip())) return code_triples def read_gitignore(path): gitignore_path = os.path.join(path, '.gitignore') patterns = [] if os.path.exists(gitignore_path): with open(gitignore_path, 'r') as file: for line in file: line = line.strip() if line and not line.startswith('#'): patterns.append(line) return patterns def is_ignored(entry_path, gitignore_patterns): for pattern in gitignore_patterns: if fnmatch.fnmatch(entry_path, pattern): return True return False def build_directory_structure(path='.', indent='', is_last=True, parent_prefix='', is_root=True): gitignore_patterns = read_gitignore(path) + [".gitignore", "*gpt_migrate/*"] if indent == '' else ["*gpt_migrate/*"] base_name = os.path.basename(path) if not base_name: base_name = '.' if indent == '': prefix = '|-- ' if not is_root else '' elif is_last: prefix = parent_prefix + '└── ' else: prefix = parent_prefix + '├── ' if os.path.isdir(path): result = indent + prefix + base_name + '/\n' if not is_root else '' else: result = indent + prefix + base_name + '\n' if os.path.isdir(path): entries = os.listdir(path) for index, entry in enumerate(entries): entry_path = os.path.join(path, entry) new_parent_prefix = ' ' if is_last else '│ ' if not is_ignored(entry_path, gitignore_patterns): result += build_directory_structure(entry_path, indent + ' ', index == len(entries) - 1, parent_prefix + new_parent_prefix, is_root=False) return result def copy_files(sourcedir, targetdir, excluded_files=[]): gitignore_patterns = read_gitignore(sourcedir) + [".gitignore"] for item in os.listdir(sourcedir): if os.path.isfile(os.path.join(sourcedir, item)): if item.endswith(INCLUDED_EXTENSIONS) and item not in excluded_files: if not is_ignored(item, gitignore_patterns): os.makedirs(targetdir, exist_ok=True) shutil.copy(os.path.join(sourcedir, item), targetdir) typer.echo(typer.style(f"Copied {item} from {sourcedir} to {targetdir}", fg=typer.colors.GREEN)) else: copy_files(os.path.join(sourcedir, item), os.path.join(targetdir, item)) def construct_relevant_files(files): ret = "" for file in files: name = file[0] content = file[1] ret += name+":\n\n" + "```\n"+content+"\n```\n\n" return ret def file_exists_in_memory(filename): file = Path('memory/' + filename) return file.exists() def convert_sigs_to_string(sigs): sig_string = "" for sig in sigs: sig_string += sig["signature"] + "\n" + sig["description"] + "\n\n" return sig_string def write_to_memory(filename,content): with open('memory/'+filename, 'a+') as file: for item in content: if item not in file.read().split("\n"): file.write(item+'\n') def read_from_memory(filename): content = "" with open('memory/'+filename, 'r') as file: content = file.read() return content def find_and_replace_file(filepath,find,replace): with open(filepath, 'r') as file: testfile_content = file.read() testfile_content = testfile_content.replace(find,replace) with open(filepath, 'w') as file: file.write(testfile_content) ================================================ FILE: pyproject.toml ================================================ [tool.poetry] name = "gpt-migrate" version = "0.1.0" description = "Easily migrate your codebase from one framework or language to another." authors = ["0xpayne"] [tool.poetry.dependencies] python = "^3.9" typer = "^0.9.0" langchain = "^0.0.238" yaspin = "^2.3.0" openai = "^0.27.8" tree-sitter = "^0.20.1" [build-system] requires = ["poetry-core>=1.0.0"] build-backend = "poetry.core.masonry.api"