Full Code of prodigysml/Dr.-Watson for AI

Repository: prodigysml/Dr.-Watson
Branch: master
Commit: c52e0464256b
Files: 5
Total size: 19.2 KB

Directory structure:
gitextract_2k41eea_/

├── .gitignore
├── README.md
├── __version__.py
├── dr-watson.py
└── issues_library.json

================================================
FILE CONTENTS
================================================

================================================
FILE: .gitignore
================================================
# Byte-compiled / optimized / DLL files
__pycache__/
*.py[cod]
*$py.class

# C extensions
*.so

# Distribution / packaging
.Python
build/
develop-eggs/
dist/
downloads/
eggs/
.eggs/
lib/
lib64/
parts/
sdist/
var/
wheels/
*.egg-info/
.installed.cfg
*.egg
MANIFEST

# PyInstaller
#  Usually these files are written by a python script from a template
#  before PyInstaller builds the exe, so as to inject date/other infos into it.
*.manifest
*.spec

# Installer logs
pip-log.txt
pip-delete-this-directory.txt

# Unit test / coverage reports
htmlcov/
.tox/
.coverage
.coverage.*
.cache
nosetests.xml
coverage.xml
*.cover
.hypothesis/
.pytest_cache/

# Translations
*.mo
*.pot

# Django stuff:
*.log
local_settings.py
db.sqlite3

# Flask stuff:
instance/
.webassets-cache

# Scrapy stuff:
.scrapy

# Sphinx documentation
docs/_build/

# PyBuilder
target/

# Jupyter Notebook
.ipynb_checkpoints

# pyenv
.python-version

# celery beat schedule file
celerybeat-schedule

# SageMath parsed files
*.sage.py

# Environments
.env
.venv
env/
venv/
ENV/
env.bak/
venv.bak/

# Spyder project settings
.spyderproject
.spyproject

# Rope project settings
.ropeproject

# mkdocs documentation
/site

# mypy
.mypy_cache/


================================================
FILE: README.md
================================================
# Dr. Watson

Dr. Watson is a simple Burp Suite extension that helps find assets, keys, subdomains, IP addresses, and other useful information! It's your very own discovery side kick, the Dr. Watson to your Sherlock! 

[![License](https://img.shields.io/badge/license-GPL3-_red.svg)](https://www.gnu.org/licenses/gpl-3.0.en.html) [![Twitter](https://img.shields.io/badge/twitter-@sml555__-blue.svg)](https://twitter.com/sml555_) ![Version](https://img.shields.io/badge/version-1.0.1-blue.svg)

# How Does Dr. Watson Work?

Dr. Watson takes regexes from the issues_library.json file and attempts to match said regexes with responses within Burp Suite. Once it matches a regex, it raises an issue with the severity defined in the config, as a finding for the target host. It is simple, sweet, and easy to use! 

# Setup - Installing for Burp Suite Pro
## Setting Up Jython
1. Download the latest standalone version of [jython](https://www.jython.org/download)
2. Navigate to Extender -> Options
3. Navigate to the "Python Environment" section
4. Click "Select File" and select the previously downloaded file

## Installing the Plugin
1. Navigate to Extender -> Extensions
2. Click the "Add" button
3. Change the "Extension Type" to "Python"
4. Select the plugin python file within the "Extension file" field
5. Click "Next"
6. Enjoy the plugin!

# How to Use The Plugin

1. Install the plugin
2. Add any domain you want analysed into scope (if not in scope, it will not be analysed, ensuring performance is not hindered immensely)
3. Navigate / crawl through the website and observe the plugin creates issues for different resources identified. 

# Authors and Thanks
Originally written by Sajeeb Lohani ([sml555](https://twitter.com/sml555_)). I would like to thank the following for helping with the project:
* BugCrowd HUNT for the Jython installation steps
* Redhunt Labs for the original plugin and the idea
* TruffleHog Regexes and git-all-secrets for the regexes

# Contributions
Contributions to this project are very welcome. If you're a newcomer to open source and would like some help in doing so, feel free to reach out to me on twitter ([@sml555_](https://twitter.com/sml555_)) and I'll assist wherever I can. 


================================================
FILE: __version__.py
================================================
__version__ = "1.0.1"

================================================
FILE: dr-watson.py
================================================
"""
Dr. Watson: Burp Suite Extension that helps you find assets, keys, and useful information. Your very own Burp side kick!
By: Sajeeb Lohani (sml555)
Twitter: https://twitter.com/sml555_

todo: fix dups better, add in api secrets, add in more JS parsing

Code Credits:
Redhunt Labs for making the original asset discovery plugin
OpenSecurityResearch CustomPassiveScanner: https://github.com/OpenSecurityResearch/CustomPassiveScanner
PortSwigger example-scanner-checks: https://github.com/PortSwigger/example-scanner-checks
"""

from burp import IBurpExtender
from burp import IScannerCheck
from burp import IScanIssue
from array import array
import re
import json
import __version__


class BurpExtender(IBurpExtender, IScannerCheck):
    """
    Implement BurpExtender to inherit from multiple base classes
    IBurpExtender is the base class required for all extensions
    IScannerCheck lets us register our extension with Burp as a custom scanner check
    """

    def registerExtenderCallbacks(self, callbacks):
        """
        The only method of the IBurpExtender interface.
        This method is invoked when the extension is loaded and registers
        an instance of the IBurpExtenderCallbacks interface
        """

        # Put the callbacks parameter into a class variable so we have class-level scope
        self._callbacks = callbacks

        # Set the name of our extension, which will appear in the Extender tool when loaded
        self._callbacks.setExtensionName("Dr. Watson")

        # Register our extension as a custom scanner check, so Burp will use this extension
        # to perform active or passive scanning and report on scan issues returned
        self._callbacks.registerScannerCheck(self)

        library_file = open("issues_library.json")
        library_file = library_file.read()

        self.library = json.loads(library_file)

        return

    def consolidateDuplicateIssues(self, existingIssue, newIssue):
        """
        This method is called when multiple issues are reported for the same URL
        In this case we are checking if the issue detail is different, as the
        issues from our scans include affected parameters/values in the detail,
        which we will want to report as unique issue instances
        """
        if existingIssue.getIssueDetail() == newIssue.getIssueDetail():
            return -1
        return 0

    def doPassiveScan(self, baseRequestResponse):
        """
        Implement the doPassiveScan method of IScannerCheck interface
        Burp Scanner invokes this method for each base request/response that is passively scanned.
        """
        # Local variables used to store a list of ScanIssue objects
        scan_issues = list()

        # Create an instance of our CustomScans object, passing the
        # base request and response, and our callbacks object
        self._CustomScans = CustomScans(baseRequestResponse, self._callbacks)

        mime_type = str(self._CustomScans.response.getStatedMimeType())

        # purposely not including SVG in case any interesting information is found
        image_types = ["GIF", "JPEG", "PNG", "image"]

        if mime_type in image_types:
            return None 
        print(mime_type)

        for issue in self.library:
            scan_issues += self._CustomScans.findRegEx(*issue[:4])

        # Finally, per the interface contract, doPassiveScan needs to return a
        # list of scan issues, if any, and None otherwise
        if scan_issues:
            return scan_issues
        else:
            return None


class CustomScans:
    unique_list = dict()
    regexes_compiled = dict()

    def __init__(self, requestResponse, callbacks):
        # Set class variables with the arguments passed to the constructor
        self._requestResponse = requestResponse
        self._callbacks = callbacks

        # Get an instance of IHelpers, which has lots of useful methods, as a class
        # variable, so we have class-level scope to all the helper methods
        self._helpers = self._callbacks.getHelpers()

        self.response = self._helpers.analyzeResponse(requestResponse.getResponse())

        # Put the parameters from the HTTP message in a class variable so we have class-level scope
        self._params = self._helpers.analyzeRequest(requestResponse.getRequest()).getParameters()
        return

    def findRegEx(self, regex, issuename, issuelevel, issuedetail):
        """
        This is a custom scan method to Look for all occurrences in the response
        that match the passed regular expression
        """
        scan_issues = []
        offset = array('i', [0, 0])
        response = self._requestResponse.getResponse()
        response_length = len(response)

        # Only check responses for 'in scope' URLs

        if self._callbacks.isInScope(self._helpers.analyzeRequest(self._requestResponse).getUrl()):

            # Compile the regular expression, telling Python to ignore EOL/LF
            # NOTE: testing required significantly here!
            if regex in CustomScans.regexes_compiled:
                myre = CustomScans.regexes_compiled[regex]
            else:
                myre = re.compile(regex, re.DOTALL)
                CustomScans.regexes_compiled[regex] = myre

            # Using the regular expression, find all occurrences in the base response
            match_vals = myre.findall(self._helpers.bytesToString(response))

            for ref in match_vals:
                url = self._helpers.analyzeRequest(self._requestResponse).getUrl()

                # Don't add the source domain to issues
                if ref.split("//")[-1].split("/")[0].split('?')[0].split(':')[0] == str(url).split("//")[-1].split(":")[0].split('?')[0]:
                    continue

                # For each matched value found, find its start position, so that we can create
                # the offset needed to apply appropriate markers in the resulting Scanner issue
                offsets = []
                start = self._helpers.indexOf(response,
                                              ref, True, 0, response_length)
                offset[0] = start
                offset[1] = start + len(ref)
                offsets.append(offset)

                base_url = str(url).split("//")[-1].split("/")[0].split('?')[0].split(":")[0]

                # Create a ScanIssue object and append it to our list of issues, marking
                # the matched value in the response.

                # create individual classes per unique asset class

                if issuename == "Asset Discovered: Domain":
                    ref = ref.split("//")[-1].split("/")[0].split('?')[0]
                    if ref.endswith("." + self._get_core_domain(url)):
                        continue

                elif issuename == "Asset Discovered: IP":
                    ref_array = ref.split(".")
                    must_continue = False
                    for s in ref_array:
                        # checks to see if 0 is in front of the ip number, 
                        # > 255 or < 0
                        i = int(s)
                        if not (0 < i < 255) or i or s != str(i):
                            must_continue = True
                            break

                    if ref_array[0] == "0":
                        continue

                    if must_continue:
                        continue

                elif issuename == "Asset Discovered: Subdomain":
                    ref = ref.split("//")[-1].split("/")[0].split('?')[0]
                    coredomain = self._get_core_domain(url)
                    if not ref.endswith("." + coredomain) or ref == coredomain:
                        continue

                elif issuename == "Asset Discovered: S3 Bucket":
                    try:
                        # getting the S3 bucket name and catch exception if regex catches incorrect data
                        ref = ref.split(" ")[0].split('/')[2]
                    except:
                        continue
                elif issuename == "Asset Discovered: DigitalOcean Space":
                    ref = ref.split('/')[2]

                elif issuename == "Asset Discovered: Azure Blob":
                    ref = ref.split(" ")[0].split('/')[2] + ":" + ref.split(" ")[0].split('/')[3]

                # this was done to only keep a single issue created for each ref
                if not self.check_unique(base_url, ref):
                    continue

                scan_issues.append(ScanIssue(self._requestResponse.getHttpService(),
                    self._helpers.analyzeRequest(self._requestResponse).getUrl(),
                    [self._callbacks.applyMarkers(self._requestResponse, None, offsets)],
                    issuename, issuelevel, issuedetail.replace("$asset$", ref)))

        return scan_issues

    def _get_core_domain(self, url):
        domain = str(url).split("//")[-1].split(":")[0].split('?')[0]
        return str(domain).rsplit('.')[-2]+"."+str(domain).rsplit('.')[-1]

    def check_unique(self, core, ref):
        if core in CustomScans.unique_list.keys():
            if ref in CustomScans.unique_list[core]:
                return False
            else:
                return True
        else:
            CustomScans.unique_list[core] = [ref]
            return True


class ScanIssue(IScanIssue):
    """
    Implementation of the IScanIssue interface with simple constructor and getter methods
    """

    def __init__(self, httpservice, url, requestresponsearray, name, severity, detailmsg):
        self._url = url
        self._httpservice = httpservice
        self._requestresponsearray = requestresponsearray
        self._name = name
        self._severity = severity
        self._detailmsg = detailmsg

    def getUrl(self):
        return self._url

    def getHttpMessages(self):
        return self._requestresponsearray

    def getHttpService(self):
        return self._httpservice

    def getRemediationDetail(self):
        return None

    def getIssueDetail(self):
        return self._detailmsg

    def getIssueBackground(self):
        return None

    def getRemediationBackground(self):
        return None

    def getIssueType(self):
        return 0

    def getIssueName(self):
        return self._name

    def getSeverity(self):
        return self._severity

    def getConfidence(self):
        return "Tentative"


================================================
FILE: issues_library.json
================================================
[
    [
        "(?:[\\d]{1,3})\\.(?:[\\d]{1,3})\\.(?:[\\d]{1,3})\\.(?:[\\d]{1,3})",
        "Asset Discovered: IP",
        "Information",
        "IP Discovered: <b>$asset$</b><br><br><b>Note:</b> Before performing any active assessment of the identified asset, please check with the owner. The asset might not be owned by the same owner/organizaion or part of the scope."
    ],
    [
        "http[s]?://(?:[a-zA-Z]|[0-9]|[$-_@.&+]|[!*\\(\\),]|(?:%[0-9a-fA-F][0-9a-fA-F]))[^><'\" \n)]+",
        "Asset Discovered: Domain",
        "Information",
        "Domain Discovered: <b>$asset$</b><br><br><b>Note:</b> Before performing any active assessment of the identified asset, please check with the owner. The asset might not be owned by the same owner/organizaion or part of the scope."
    ],
    [
        "http[s]?://(?:[a-zA-Z]|[0-9]|[$-_@.&+]|[!*\\(\\),]|(?:%[0-9a-fA-F][0-9a-fA-F]))[^><'\" \n)]+",
        "Asset Discovered: Subdomain",
        "Information",
        "Subdomain Discovered: <b>$asset$</b><br><br><b>Note:</b> Before performing any active assessment of the identified asset, please check with the owner. The asset might not be owned by the same owner/organizaion or part of the scope."
    ],
    [
        "(http(?:s)?://.[^><\\'\\\" \\n\\)]+.s3.amazonaws.com|\\))",
        "Asset Discovered: S3 Bucket",
        "Information",
        "S3 Bucket Discovered: <b>$asset$</b><br><br><b>Note:</b> Before performing any active assessment of the identified asset, please check with the owner. The asset might not be owned by the same owner/organizaion or part of the scope."
    ],
    [
        "http(?:s)://[^><\\.'\" \n\\)]+.[^><\\.'\" \n\\)]+.[^><\\.'\" \n\\)]+.digitaloceanspaces.com",
        "Asset Discovered: DigitalOcean Space",
        "Information",
        "DigitalOcean Space Discovered: <b>$asset$</b><br><br><b>Note:</b> Before performing any active assessment of the identified asset, please check with the owner. The asset might not be owned by the same owner/organizaion or part of the scope."
    ],
    [
        "http(?:s)://.[^><'\" \n\\)]+.blob.core.windows.net/.[^><'\" \n/)]+./",
        "Asset Discovered: Azure Blob",
        "Information",
        "Azure Blob Discovered: <b>$asset$</b><br><br><b>Note:</b> Before performing any active assessment of the identified asset, please check with the owner. The asset might not be owned by the same owner/organizaion or part of the scope."
    ],
    [
        "-----BEGIN RSA PRIVATE KEY-----",
        "Asset Discovered: RSA Private Key",
        "High",
        "RSA Private Key Discovered: <b>$asset$</b><br><br><b>Note:</b> Before performing any active assessment of the identified asset, please check with the owner. The asset might not be owned by the same owner/organizaion or part of the scope."
    ],
    [
        "-----BEGIN DSA PRIVATE KEY-----",
        "Asset Discovered: DSA Private Key",
        "High",
        "DSA Private Key Discovered: <b>$asset$</b><br><br><b>Note:</b> Before performing any active assessment of the identified asset, please check with the owner. The asset might not be owned by the same owner/organizaion or part of the scope."
    ],
    [
        "-----BEGIN EC PRIVATE KEY-----",
        "Asset Discovered: EC Private Key",
        "High",
        "EC Private Key Discovered: <b>$asset$</b><br><br><b>Note:</b> Before performing any active assessment of the identified asset, please check with the owner. The asset might not be owned by the same owner/organizaion or part of the scope."
    ],
    [
        "-----BEGIN PGP PRIVATE KEY BLOCK-----",
        "Asset Discovered: PGP Private Key",
        "High",
        "PGP Private Key Discovered: <b>$asset$</b><br><br><b>Note:</b> Before performing any active assessment of the identified asset, please check with the owner. The asset might not be owned by the same owner/organizaion or part of the scope."
    ],
    [
        "X-Octopus-ApiKey",
        "Asset Discovered: Octopus API Key",
        "Medium",
        "Octopus API Key Discovered: <b>$asset$</b><br><br><b>Note:</b> Before performing any active assessment of the identified asset, please check with the owner. The asset might not be owned by the same owner/organizaion or part of the scope."
    ],
    [
        "X-NuGet-ApiKey",
        "Asset Discovered: NuGet API Key",
        "Medium",
        "NuGet API Key Discovered: <b>$asset$</b><br><br><b>Note:</b> Before performing any active assessment of the identified asset, please check with the owner. The asset might not be owned by the same owner/organizaion or part of the scope."
    ],
    [
        "(xox[p|b|o|a]-[0-9]{12}-[0-9]{12}-[0-9]{12}-[a-z0-9]{32})",
        "Asset Discovered: Slack token",
        "Medium",
        "Slack token Discovered: <b>$asset$</b><br><br><b>Note:</b> Before performing any active assessment of the identified asset, please check with the owner. The asset might not be owned by the same owner/organizaion or part of the scope."
    ],
    [
        "(AccessKeyId|aws_access_key_id)",
        "Asset Discovered: AWS Access Key ID",
        "High",
        "AWS Access Key ID Discovered: <b>$asset$</b><br><br><b>Note:</b> Before performing any active assessment of the identified asset, please check with the owner. The asset might not be owned by the same owner/organizaion or part of the scope."
    ],
    [
        "(SecretAccessKey|aws_secret_access_key)",
        "Asset Discovered: AWS Secret Access Key ID",
        "High",
        "AWS Secret Access Key ID Discovered: <b>$asset$</b><br><br><b>Note:</b> Before performing any active assessment of the identified asset, please check with the owner. The asset might not be owned by the same owner/organizaion or part of the scope."
    ]
]