[ { "path": ".github/workflows/go.yml", "content": "name: Go\n\non:\n push:\n branches: [ main ]\n pull_request:\n branches: [ main ]\n\njobs:\n build:\n name: Build and Test\n runs-on: ubuntu-latest\n \n steps:\n - uses: actions/checkout@v3\n\n - name: Set up Go\n uses: actions/setup-go@v4\n with:\n go-version: '1.21'\n cache: true\n \n - name: Install dependencies\n run: cd go && go mod download\n\n - name: Run golangci-lint\n uses: golangci/golangci-lint-action@v3\n with:\n version: latest\n working-directory: go\n args: --timeout=5m\n\n - name: Run tests\n run: cd go && go test -v ./... -coverprofile=coverage.txt -covermode=atomic\n\n - name: Upload coverage to Codecov\n uses: codecov/codecov-action@v3\n with:\n file: ./go/coverage.txt\n flags: unittests\n\n - name: Build\n run: cd go && go build -v ./cmd/movery\n\n release:\n name: Create Release\n needs: build\n runs-on: ubuntu-latest\n if: startsWith(github.ref, 'refs/tags/')\n \n steps:\n - uses: actions/checkout@v3\n\n - name: Set up Go\n uses: actions/setup-go@v4\n with:\n go-version: '1.21'\n\n - name: Build for multiple platforms\n run: |\n cd go\n GOOS=linux GOARCH=amd64 go build -o movery-linux-amd64 ./cmd/movery\n GOOS=windows GOARCH=amd64 go build -o movery-windows-amd64.exe ./cmd/movery\n GOOS=darwin GOARCH=amd64 go build -o movery-darwin-amd64 ./cmd/movery\n\n - name: Create Release\n uses: softprops/action-gh-release@v1\n with:\n files: |\n go/movery-linux-amd64\n go/movery-windows-amd64.exe\n go/movery-darwin-amd64\n env:\n GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} " }, { "path": ".gitignore", "content": "# Python\n__pycache__/\n*.py[cod]\n*$py.class\n*.so\n.Python\nbuild/\ndevelop-eggs/\ndist/\ndownloads/\neggs/\n.eggs/\nlib/\nlib64/\nparts/\nsdist/\nvar/\nwheels/\n*.egg-info/\n.installed.cfg\n*.egg\n\n# Go\n*.exe\n*.exe~\n*.dll\n*.so\n*.dylib\n*.test\n*.out\ngo.work\n/go/bin/\n/go/pkg/\n\n# IDE\n.idea/\n.vscode/\n*.swp\n*.swo\n\n# Project specific\n.cache/\nreports/\n*.log\nprofile.stats\n.coverage\nhtmlcov/\n\n# Environment\n.env\n.venv\nenv/\nvenv/\nENV/\n\n# OS\n.DS_Store\nThumbs.db \n\n# dataset\ndataset/\n" }, { "path": "CODE_OF_CONDUCT.md", "content": "# 贡献者行为准则\n\n## 我们的承诺\n\n为了营造一个开放和友好的环境，我们作为贡献者和维护者承诺：无论年龄、体型、身体健全与否、民族、性征、性别认同和表达、经验水平、教育程度、社会地位、国籍、相貌、种族、宗教信仰、性取向如何，我们都会确保每个参与项目的人都不受骚扰。\n\n## 我们的标准\n\n有助于创造积极环境的行为包括：\n\n* 使用友好和包容的语言\n* 尊重不同的观点和经验\n* 优雅地接受建设性批评\n* 关注对社区最有利的事情\n* 友善对待其他社区成员\n\n不当行为包括：\n\n* 使用带有性色彩的语言或图像，以及不受欢迎的性关注或advances\n* 发表挑衅、侮辱/贬损的评论，进行人身攻击或政治攻击\n* 公开或私下骚扰\n* 未经明确许可，发布他人的私人信息，如物理或电子地址\n* 其他可以被合理地认定为不恰当或违反职业操守的行为\n\n## 我们的责任\n\n项目维护者有责任为可接受的行为标准做出诠释，并采取恰当且公平的纠正措施来应对任何不可接受的行为。\n\n项目维护者有权利和责任删除、编辑或拒绝违反本行为准则的评论、提交、代码、wiki编辑、问题和其他贡献，并暂时或永久地禁止任何他们认为不当、威胁、冒犯或有害的行为的贡献者。\n\n## 范围\n\n当一个人代表项目或其社区时，本行为准则适用于项目空间和公共空间。代表项目或社区的示例包括使用官方项目电子邮件地址、通过官方社交媒体账户发布，或在线上或线下活动中担任指定代表。项目的代表性可由项目维护者进一步定义和澄清。\n\n## 强制执行\n\n可以通过[在此处插入联系方式]向项目团队报告辱骂、骚扰或其他不可接受的行为。所有投诉都将得到审查和调查，并将导致做出适当且必要的回应。项目团队有义务对事件报告者保密。具体执行政策的更多细节可能会单独发布。\n\n不遵守或不执行本行为准则的项目维护者可能会因项目领导层的决定而暂时或永久地失去其在项目中的角色。\n\n## 归属\n\n本行为准则改编自[贡献者公约][homepage]，版本1.4，可在[http://contributor-covenant.org/version/1/4][version]查看。\n\n[homepage]: http://contributor-covenant.org\n[version]: http://contributor-covenant.org/version/1/4/ " }, { "path": "CONTRIBUTING.md", "content": "# 贡献指南\n\n感谢您对Re-movery项目的关注！我们欢迎任何形式的贡献，包括但不限于：\n\n- 报告问题\n- 提交功能建议\n- 改进文档\n- 提交代码修复\n- 添加新功能\n\n## 开发环境设置\n\n1. 安装Go 1.21或更高版本\n2. 克隆仓库：\n ```bash\n git clone https://github.com/heyangxu/Re-movery.git\n cd Re-movery\n ```\n3. 安装依赖：\n ```bash\n cd go\n go mod download\n ```\n4. 安装开发工具：\n ```bash\n # 安装golangci-lint\n go install github.com/golangci/golangci-lint/cmd/golangci-lint@latest\n ```\n\n## 开发流程\n\n1. 创建新分支：\n ```bash\n git checkout -b feature/your-feature-name\n ```\n\n2. 进行开发，确保：\n - 遵循Go代码规范\n - 添加适当的测试\n - 更新相关文档\n\n3. 运行测试：\n ```bash\n make test\n ```\n\n4. 运行代码检查：\n ```bash\n make lint\n ```\n\n5. 提交代码：\n ```bash\n git add .\n git commit -m \"feat: Add your feature description\"\n ```\n\n6. 推送到GitHub：\n ```bash\n git push origin feature/your-feature-name\n ```\n\n7. 创建Pull Request\n\n## 提交规范\n\n我们使用[Conventional Commits](https://www.conventionalcommits.org/)规范，提交信息格式如下：\n\n```\n(): \n\n[optional body]\n\n[optional footer]\n```\n\n类型（type）包括：\n- feat: 新功能\n- fix: 修复\n- docs: 文档更新\n- style: 代码格式（不影响代码运行的变动）\n- refactor: 重构\n- perf: 性能优化\n- test: 测试\n- chore: 构建过程或辅助工具的变动\n\n## 代码规范\n\n- 遵循[Go代码规范](https://golang.org/doc/effective_go)\n- 使用`gofmt`格式化代码\n- 添加适当的注释\n- 保持代码简洁明了\n- 使用有意义的变量和函数名\n\n## 测试规范\n\n- 为新功能添加单元测试\n- 确保测试覆盖率不降低\n- 测试应该简单明了\n- 避免测试之间的依赖\n\n## 文档规范\n\n- 保持README.md的更新\n- 为新功能添加文档\n- 更新API文档\n- 添加示例代码\n\n## 问题反馈\n\n如果您发现了问题或有新的想法，请：\n\n1. 检查是否已存在相关的Issue\n2. 如果没有，创建新的Issue\n3. 清晰描述问题或建议\n4. 提供复现步骤（如果适用）\n5. 提供相关的日志或截图（如果适用）\n\n## 行为准则\n\n请参阅我们的[行为准则](CODE_OF_CONDUCT.md)。\n\n## 许可证\n\n通过提交代码，您同意您的代码遵循项目的[MIT许可证](LICENSE)。 " }, { "path": "Detector.py", "content": "\"\"\"\nMOVERY Detector - 漏洞检测系统\n作者: Seunghoon Woo (seunghoonwoo@korea.ac.kr)\n修改: August 5, 2022\n\n主要功能:\n1. 扫描目标程序中的潜在漏洞\n2. 支持抽象和非抽象两种匹配模式\n3. 基于代码相似度的漏洞检测\n4. 支持多种漏洞特征匹配方式\n\"\"\"\n\n# 导入必要的库\nimport os\nimport sys\ncurrentPath = os.getcwd()\nsys.path.append(currentPath + \"/config/\")\nimport movery_config\nimport subprocess\nimport re\nimport json\nimport time\n\n\"\"\"全局变量\"\"\"\ndelimiter = \"\\r\\0?\\r?\\0\\r\" # 用于分隔的定界符\ntheta = 0.5 # 相似度阈值\n\n\"\"\"路径配置\"\"\"\n# 漏洞特征数据集路径\nvulESSLinePath = currentPath + \"/dataset/vulESSLines/\" # 漏洞必要行路径\nvulDEPLinePath = currentPath + \"/dataset/vulDEPLines/\" # 漏洞依赖行路径\nnoOldESSLinePath = currentPath + \"/dataset/noOldESSLines/\" # 无旧版本必要行路径\nnoOldDEPLinePath = currentPath + \"/dataset/noOldDEPLines/\" # 无旧版本依赖行路径\npatESSLinePath = currentPath + \"/dataset/patESSLines/\" # 补丁必要行路径\npatDEPLinePath = currentPath + \"/dataset/patDEPLines/\" # 补丁依赖行路径\nvulBodyPath = currentPath + \"/dataset/vulBodySet/\" # 漏洞函数体集合路径\nvulHashPath = currentPath + \"/dataset/vulHashes/\" # 漏洞哈希值路径\ntargetPath = currentPath + \"/dataset/tarFuncs/\" # 目标函数路径\nossidxPath = currentPath + \"/dataset/oss_idx.txt\" # OSS索引文件路径\nidx2verPath = currentPath + \"/dataset/idx2cve.txt\" # CVE索引文件路径\n\n\"\"\"工具函数\"\"\"\ndef intersect(a, b):\n \"\"\"计算两个列表的交集\"\"\"\n return list(set(a) & set(b))\n\ndef union(a, b):\n \"\"\"计算两个列表的并集\"\"\"\n return list(set(a) | set(b))\n\ndef jaccard_sim(a, b):\n \"\"\"计算Jaccard相似度: 交集大小/并集大小\"\"\"\n inter = len(list(set(a).intersection(b)))\n union = (len(set(a)) + len(b)) - inter\n return float(inter) / union\n\ndef normalize(string):\n \"\"\"\n 标准化字符串:\n - 移除回车符和制表符\n - 移除所有空格\n - 转换为小写\n 参考: https://github.com/squizz617/vuddy\n \"\"\"\n return ''.join(string.replace('\\r', '').replace('\\t', '').split(' ')).lower()\n\ndef removeComment(string):\n \"\"\"\n 移除C/C++风格的注释\n 支持:\n - 单行注释 (//)\n - 多行注释 (/* */)\n 参考: https://github.com/squizz617/vuddy\n \"\"\"\n c_regex = re.compile(\n r'(?P//.*?$|[{}]+)|(?P/\\*.*?\\*/)|(?P\\'(\\\\.|[^\\\\\\'])*\\'|\"(\\\\.|[^\\\\\"])*\"|.[^/\\'\"]*)',\n re.DOTALL | re.MULTILINE)\n return ''.join([c.group('noncomment') for c in c_regex.finditer(string) if c.group('noncomment')])\n\ndef readFile(path):\n \"\"\"\n 读取文件内容,支持多种编码:\n - UTF-8\n - CP949\n - euc-kr\n \"\"\"\n body = ''\n try:\n fp = open(path, 'r', encoding = \"UTF-8\")\n body = ''.join(fp.readlines()).strip()\n except:\n try:\n fp = open(path, 'r', encoding = \"CP949\")\n body = ''.join(fp.readlines()).strip()\n except:\n try:\n fp = open(path, 'r', encoding = \"euc-kr\")\n body = ''.join(fp.readlines()).strip()\n except:\n pass\n return body\n\ndef readOSSIDX():\n \"\"\"读取OSS索引文件,构建OSS索引字典\"\"\"\n ossIDX = {}\n with open(ossidxPath, 'r', encoding = \"UTF-8\") as foss:\n body = ''.join(foss.readlines()).strip()\n for each in body.split('\\n'):\n if each.split('@@')[0] not in ossIDX:\n ossIDX[each.split('@@')[0]] = []\n ossIDX[each.split('@@')[0]].append(each.split('@@')[1])\n return ossIDX\n\ndef readIDX2VER():\n \"\"\"读取CVE索引文件,构建CVE版本映射\"\"\"\n idx2ver = {}\n with open(idx2verPath, 'r', encoding = \"UTF-8\") as idxfp:\n body = ''.join(idxfp.readlines()).strip()\n for each in body.split('\\n'):\n idx2ver[each.split('##')[0]] = (each.split('##')[1])\n return idx2ver\n\ndef readVulHashes():\n \"\"\"读取漏洞哈希值文件,构建漏洞哈希字典\"\"\"\n vulHashes = {}\n for files in os.listdir(vulHashPath):\n oss = files.split('_hash.txt')[0]\n vulHashes[oss] = []\n\n with open(vulHashPath+ files, 'r', encoding = \"UTF-8\") as fo:\n body = ''.join(fo.readlines()).strip()\n for each in body.split('\\n'):\n hashval = each.split('\\t')[0]\n vulHashes[oss].append(hashval)\n return vulHashes\n\ndef spaceReduction(tar, vulHashes, ossIDX):\n \"\"\"\n 搜索空间规约函数\n \n 参数:\n tar: 目标程序\n vulHashes: 漏洞哈希值字典\n ossIDX: OSS索引字典\n \n 返回:\n tarIDX: 目标索引列表\n tarFuncs: 目标函数字典\n \n 功能:\n 1. 通过哈希匹配快速筛选可能存在漏洞的函数\n 2. 减少后续详细分析的搜索空间\n \"\"\"\n funcHash = {}\n tarIDX = []\n tarFuncs = {}\n res = {}\n\n # 检查目标文件是否存在\n if not os.path.isfile(targetPath + '/' + tar + '_hash.txt') or not os.path.isfile(targetPath + '/' + tar + '_funcs.txt'):\n print (\"No tar files (tar_funcs.txt and tar_hash.txt) in './dataset/tarFuncs/'.\")\n sys.exit()\n\n # 读取目标函数哈希值\n with open(targetPath + '/' + tar + '_hash.txt', 'r', encoding = \"UTF-8\") as fh:\n body = ''.join(fh.readlines()).strip()\n for each in body.split('\\n'):\n hashval = each.split('\\t')[0]\n hashpat = each.split('\\t')[1]\n if hashval not in funcHash:\n funcHash[hashval] = []\n funcHash[hashval].append(hashpat)\n\n # 进行哈希匹配\n for oss in vulHashes:\n if oss in ossIDX:\n for hashval in vulHashes[oss]:\n if hashval in funcHash: \n tarIDX.extend(ossIDX[oss])\n for eachPat in funcHash[hashval]:\n res['@@'.join(eachPat.split('##')[1].split('@@')[:-1])] = 1\n\n tarIDX = list(set(tarIDX))\n\n # 读取目标函数\n with open(targetPath + tar + '_funcs.txt', 'r', encoding = \"UTF-8\") as ft:\n tarFuncs = json.load(ft)\n\n # 筛选相关函数\n tempTar = {}\n for file in tarFuncs:\n if ('@@'.join(file.split('##')[1].split('@@')[:-1])) in res:\n tempTar[file] = tarFuncs[file]\n \n tarFuncs = tempTar\n return tarIDX, tarFuncs\n\ndef detector(tar):\n \"\"\"\n 主要漏洞检测函数\n \n 参数:\n tar: 目标程序名称\n \n 功能:\n 1. 加载必要的索引和哈希数据\n 2. 进行空间规约\n 3. 对每个潜在漏洞进行检测:\n - 检查必要代码行\n - 检查依赖关系\n - 计算相似度\n - 应用抽象或非抽象匹配\n \"\"\"\n print ()\n print (\"[+] NOW MOVERY SCANS \" + tar + \"...\")\n print ()\n\n # 计时开始\n mtime = 0.0 \n\n # 读取必要数据\n ossIDX = readOSSIDX()\n idx2ver = readIDX2VER()\n vulHashes = readVulHashes()\n tarIDX, tarFuncs = spaceReduction(tar, vulHashes, ossIDX)\n\n # 对每个漏洞文件进行检测\n for vulFiles in os.listdir(vulBodyPath):\n temp = {}\n idx = vulFiles.split('_')[0]\n\n # 仅考虑目标程序中重用的OSS\n if idx not in tarIDX:\n continue\n\n vulBody = \"\"\n\n # 初始化各类特征行\n vul_essLines = [] # 漏洞必要行\n vul_depLines = {} # 漏洞依赖行\n pat_essLines = [] # 补丁必要行\n pat_depLines = {} # 补丁依赖行\n\n flag = 0 # 标记漏洞类型\n isAbs = 1 # 是否使用抽象匹配\n\n # 读取漏洞信息\n with open(vulBodyPath + vulFiles, 'r', encoding = \"UTF-8\") as f:\n vulBody = json.load(f)\n \n # 处理不同类型的漏洞特征\n if idx + \"_common.txt\" in os.listdir(vulESSLinePath):\n # 存在最老的漏洞函数且补丁删除了部分代码\n with open(vulESSLinePath + idx + \"_common.txt\", 'r', encoding = \"UTF-8\") as f:\n vul_essLines = json.load(f)\n with open(vulDEPLinePath + idx + \"_depen.txt\", 'r', encoding = \"UTF-8\") as fd:\n vul_depLines = json.load(fd)\n flag = 1\n\n elif idx + \"_minus.txt\" in os.listdir(noOldESSLinePath):\n # 不存在最老的漏洞函数且补丁删除了部分代码\n with open(noOldESSLinePath + idx + \"_minus.txt\", 'r', encoding = \"UTF-8\") as f:\n vul_essLines = json.load(f)\n with open(noOldDEPLinePath + idx + \"_depen.txt\", 'r', encoding = \"UTF-8\") as fd:\n vul_depLines = json.load(fd)\n flag = 1\n\n if idx + \"_plus.txt\" in os.listdir(patESSLinePath):\n # 补丁特征\n with open(patESSLinePath + idx + \"_plus.txt\", 'r', encoding = \"UTF-8\") as f:\n pat_essLines = json.load(f)\n with open(patDEPLinePath + idx + \"_depen.txt\", 'r', encoding = \"UTF-8\") as fd:\n pat_depLines = json.load(fd)\n flag = 2\n else:\n if len(vul_essLines) == 0:\n continue\n\n # 漏洞类型标记:\n # del o add x 1 - 只有删除\n # del o add o 2 - 既有删除也有添加\n # del x add o 3 - 只有添加\n\n # 选择性抽象处理\n if len(pat_essLines) > 0:\n patLines = [] # 补丁行\n patAbsLines = [] # 抽象补丁行\n vulLines = [] # 漏洞行\n vulAbsLines = [] # 抽象漏洞行\n tempNewPat = [] # 临时新补丁\n tempNewAbsPat = [] # 临时新抽象补丁\n\n # 处理补丁行\n for eachPat in pat_essLines:\n patLines.append(normalize(eachPat['pat_body']))\n patAbsLines.append(normalize(eachPat['abs_body']))\n\n if normalize(eachPat['pat_body']) not in vulBody['vul_body']:\n tempNewPat.append(normalize(eachPat['pat_body']))\n tempNewAbsPat.append(normalize(eachPat['abs_body']))\n \n # 清理特殊字符\n temp = []\n temp[:] = (value for value in tempNewPat if value != '{' and value != '}' and value != '')\n newPat = set(temp)\n\n temp[:] = (value for value in tempNewAbsPat if value != '{' and value != '}' and value != '')\n newAbsPat = set(temp)\n\n # 处理漏洞行\n if len(vul_essLines) > 0:\n for eachVul in vul_essLines:\n vulLines.append(normalize(eachVul['vul_body']))\n vulAbsLines.append(normalize(eachVul['abs_body']))\n if (set(patAbsLines) != set(vulAbsLines)): # 应用抽象\n isAbs = 1\n else:\n isAbs = 0\n else:\n flag = 3\n\n # 处理依赖行\n if len(vul_depLines) > 0:\n if \"vul\" in vul_depLines:\n vulDepens = vul_depLines[\"vul\"]\n else:\n vulDepens = vul_depLines\n\n # 初始化依赖行集合\n absDepens_withoutOLD = [] # 无旧版本抽象依赖\n norDepens_withoutOLD = [] # 无旧版本标准依赖\n absDepens_withOLD = [] # 有旧版本抽象依赖\n norDepens_withOLD = [] # 有旧版本标准依赖\n\n # 处理依赖行\n for eachDepen in vulDepens:\n if len(vulDepens[eachDepen]) > 0:\n for each in vulDepens[eachDepen]:\n absDepens_withoutOLD.append(removeComment(each[\"abs_norm_vul\"]))\n norDepens_withoutOLD.append(removeComment(each[\"orig_norm_vul\"]))\n\n # 处理旧版本依赖\n if \"old\" in vul_depLines:\n vulDepens = vul_depLines[\"old\"]\n for eachDepen in vulDepens:\n if len(vulDepens[eachDepen]) > 0:\n for each in vulDepens[eachDepen]:\n absDepens_withOLD.append(removeComment(each[\"abs_norm_vul\"]))\n norDepens_withOLD.append(removeComment(each[\"orig_norm_vul\"]))\n\n # 转换为集合\n absDepens_withoutOLD = set(absDepens_withoutOLD)\n absDepens_withOLD = set(absDepens_withOLD)\n norDepens_withoutOLD = set(norDepens_withoutOLD)\n norDepens_withOLD = set(norDepens_withOLD)\n\n # 提取核心漏洞行\n coreAbsVulLines = []\n coreVulLines = []\n\n for val in vul_essLines:\n coreAbsVulLines.append(normalize(val[\"abs_body\"]))\n coreVulLines.append(normalize(val[\"vul_body\"]))\n\n coreAbsVulLines = set(coreAbsVulLines)\n coreVulLines = set(coreVulLines)\n\n # 提取函数体集合\n vulBodySet = []\n oldBodySet = []\n\n vulBodySet = set(vulBody['vul_body'])\n if 'old_body' in vulBody:\n oldBodySet = set(vulBody['old_body'])\n\n # 对每个目标函数进行检测\n for file in tarFuncs:\n x = set(tarFuncs[file][\"norm\"]) # 标准化函数体\n y = set(tarFuncs[file][\"abst\"]) # 抽象化函数体\n \n step = 1\n\n # 处理不同类型的漏洞\n if flag == 1 or flag == 2:\n # 补丁包含添加和删除的代码行\n\n if isAbs == 1:\n # 使用抽象匹配\n\n # 检查核心漏洞行\n if not coreAbsVulLines.issubset(y):\n step = 0\n\n # 检查依赖行\n if step == 1:\n now = time.time()\n for absLine in absDepens_withoutOLD:\n if absLine not in y:\n step = 0\n break\n\n # 尝试旧版本依赖\n if step == 0 and len(absDepens_withOLD) > 0:\n step = 1\n for absLine in absDepens_withOLD:\n if absLine not in y:\n step = 0\n break\n mtime += time.time() - now\n\n # 检查补丁特征\n if step == 1 and flag == 2:\n now = time.time()\n if not newAbsPat.isdisjoint(y):\n step = 0\n mtime += time.time() - now\n\n # 计算相似度\n if step == 1:\n now = time.time()\n if len(vulBodySet) <= 3:\n continue\n\n # 检查与漏洞函数的相似度\n if float(len(vulBodySet&x)/len(vulBodySet)) >= theta:\n print ('\\t* [' + idx2ver[idx] + '] ' + tar + ' contains the vulnerable \"' + file.split('##')[0] + '\" function in ' + file.split('##')[1].replace('@@', '/'))\n continue\n mtime += time.time() - now\n\n try:\n # 检查与最老漏洞函数的相似度\n now = time.time()\n if float(len(oldBodySet&x)/len(oldBodySet)) >= theta:\n print ('\\t* [' + idx2ver[idx] + '] ' + tar + ' contains the vulnerable \"' + file.split('##')[0] + '\" function in ' + file.split('##')[1].replace('@@', '/'))\n mtime += time.time() - now\n except:\n pass\n\n else:\n # 不使用抽象匹配\n\n # 检查核心漏洞行\n now = time.time()\n if not coreVulLines.issubset(x):\n step = 0 \n mtime += time.time() - now\n\n # 检查依赖行\n if step == 1:\n now = time.time()\n for absLine in norDepens_withoutOLD:\n if absLine not in x:\n step = 0\n break\n\n # 尝试旧版本依赖\n if step == 0 and len(norDepens_withOLD) > 0:\n step = 1\n for absLine in norDepens_withOLD:\n if absLine not in x:\n step = 0\n break\n mtime += time.time() - now\n \n # 检查补丁特征\n if step == 1 and flag == 2:\n now = time.time()\n if not newPat.isdisjoint(x):\n step = 0\n mtime += time.time() - now\n\n # 计算相似度\n if step == 1:\n if len(vulBodySet) <= 3: \n continue\n now = time.time()\n if float(len(vulBodySet&x)/len(vulBodySet)) >= theta:\n print ('\\t* [' + idx2ver[idx] + '] ' + tar + ' contains the vulnerable \"' + file.split('##')[0] + '\" function in ' + file.split('##')[1].replace('@@', '/'))\n continue\n \n mtime += time.time() - now\n\n try:\n # 检查与最老漏洞函数的相似度\n now = time.time()\n if float(len(oldBodySet&x)/len(oldBodySet)) >= theta:\n print ('\\t* [' + idx2ver[idx] + '] ' + tar + ' contains the vulnerable \"' + file.split('##')[0] + '\" function in ' + file.split('##')[1].replace('@@', '/'))\n mtime += time.time() - now\n except:\n pass\n \n elif flag == 3:\n # 没有删除的代码行\n\n if (len(newAbsPat) == 0):\n continue\n\n # 检查补丁特征\n now = time.time() \n if not newAbsPat.isdisjoint(y):\n step = 0\n mtime += time.time() - now\n\n # 计算相似度\n if step == 1:\n if len(vulBodySet) <= 3: \n continue\n\n now =time.time()\n if float(len(vulBodySet&x)/len(vulBodySet)) >= theta:\n print ('\\t* [' + idx2ver[idx] + '] ' + tar + ' contains the vulnerable \"' + file.split('##')[0] + '\" function in ' + file.split('##')[1].replace('@@', '/'))\n continue\n mtime += time.time() - now\n\n try:\n # 检查与最老漏洞函数的相似度\n now = time.time()\n if float(len(oldBodySet&x)/len(oldBodySet)) >= theta:\n print ('\\t* [' + idx2ver[idx] + '] ' + tar + ' contains the vulnerable \"' + file.split('##')[0] + '\" function in ' + file.split('##')[1].replace('@@', '/'))\n mtime += time.time() - now\n except:\n pass\n\n else:\n continue\n\n print ()\n print (\"[+] TOTAL ELAPSED TIME (ONLY FOR VULNERABILITY DETECTION): \" + str(mtime) + \" s\")\n\ndef main(target):\n \"\"\"主函数,调用漏洞检测器\"\"\"\n detector(target)\n\n\"\"\"程序入口\"\"\"\nif __name__ == \"__main__\":\n # 获取命令行参数\n target = sys.argv[1] # 目标程序\n testmd = sys.argv[2] # 测试模式\n\n # 验证测试模式参数\n if testmd != '1' and testmd != '0':\n print (\"Please enter correct inputs.\")\n print (\"python3 Detector.py 'TARGET_PROGRAM' [0|1]\")\n sys.exit()\n \n # 处理测试模式\n if testmd == '1':\n # 预定义的可用目标程序列表\n currentPossible = [\"arangodb\", \"crown\", \"emscripten\", \"ffmpeg\", \"freebsd-src\", \n \"git\", \"opencv\", \"openMVG\", \"reactos\", \"redis\"]\n if target not in currentPossible:\n print (\"Please enter one of the inputs below.\")\n print (str(currentPossible))\n sys.exit()\n else:\n main(target)\n else:\n main(target)\n" }, { "path": "LICENSE", "content": "MIT License\n\nCopyright (c) 2024 heyangxu\n\nPermission is hereby granted, free of charge, to any person obtaining a copy\nof this software and associated documentation files (the \"Software\"), to deal\nin the Software without restriction, including without limitation the rights\nto use, copy, modify, merge, publish, distribute, sublicense, and/or sell\ncopies of the Software, and to permit persons to whom the Software is\nfurnished to do so, subject to the following conditions:\n\nThe above copyright notice and this permission notice shall be included in all\ncopies or substantial portions of the Software.\n\nTHE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\nIMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\nFITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\nAUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\nLIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\nOUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\nSOFTWARE. " }, { "path": "Makefile", "content": ".PHONY: build test clean lint run\n\n# Go parameters\nGOCMD=go\nGOBUILD=$(GOCMD) build\nGOCLEAN=$(GOCMD) clean\nGOTEST=$(GOCMD) test\nGOGET=$(GOCMD) get\nGOMOD=$(GOCMD) mod\nBINARY_NAME=movery\nBINARY_UNIX=$(BINARY_NAME)_unix\n\n# Build parameters\nBUILD_DIR=go/bin\nMAIN_PATH=./go/cmd/movery\n\nall: test build\n\nbuild:\n\tcd go && $(GOBUILD) -o $(BUILD_DIR)/$(BINARY_NAME) -v $(MAIN_PATH)\n\ntest:\n\tcd go && $(GOTEST) -v ./...\n\nclean:\n\tcd go && $(GOCLEAN)\n\trm -f $(BUILD_DIR)/*\n\nrun:\n\tcd go && $(GOBUILD) -o $(BUILD_DIR)/$(BINARY_NAME) -v $(MAIN_PATH)\n\t./$(BUILD_DIR)/$(BINARY_NAME)\n\nlint:\n\tcd go && golangci-lint run\n\ndeps:\n\tcd go && $(GOMOD) download\n\n# Cross compilation\nbuild-linux:\n\tcd go && CGO_ENABLED=0 GOOS=linux GOARCH=amd64 $(GOBUILD) -o $(BUILD_DIR)/$(BINARY_UNIX) -v $(MAIN_PATH)\n\nbuild-windows:\n\tcd go && CGO_ENABLED=0 GOOS=windows GOARCH=amd64 $(GOBUILD) -o $(BUILD_DIR)/$(BINARY_NAME).exe -v $(MAIN_PATH)\n\n# Help target\nhelp:\n\t@echo \"Available targets:\"\n\t@echo \" build - Build the project\"\n\t@echo \" test - Run tests\"\n\t@echo \" clean - Clean build files\"\n\t@echo \" run - Build and run the project\"\n\t@echo \" lint - Run linter\"\n\t@echo \" deps - Download dependencies\"\n\t@echo \" build-linux - Build for Linux\"\n\t@echo \" build-windows- Build for Windows\" " }, { "path": "Preprocessing.py", "content": "\"\"\"\nMOVERY 预处理器\n作者: Seunghoon Woo (seunghoonwoo@korea.ac.kr)\n修改日期: March 31, 2023.\n\n主要功能：\n1. 处理目标代码仓库中的C/C++源文件\n2. 提取所有函数并进行标准化处理\n3. 生成函数的多种表示形式(原始、标准化、抽象化)\n4. 输出处理后的函数信息和哈希值\n\n工作流程：\n1. 接收目标代码仓库路径\n2. 遍历所有C/C++源文件\n3. 使用ctags提取函数信息\n4. 对每个函数进行三重处理：\n - 保存原始代码\n - 生成标准化版本\n - 生成抽象化版本\n5. 输出处理结果：\n - 函数信息JSON文件\n - 函数哈希值文件\n\"\"\"\n\n# 导入必要的库\nimport os\nimport sys\ncurrentPath = os.getcwd()\nsys.path.append(currentPath + \"/config/\") # 添加配置文件路径\nimport subprocess # 用于执行ctags命令\nimport re # 用于正则表达式处理\nimport json # 用于JSON数据处理\nimport time\nfrom hashlib import md5 # 用于生成哈希值\n\n\"\"\"全局变量配置\"\"\"\n# 支持的源代码文件扩展名\npossible = (\".c\", \".cc\", \".cpp\") \n# 用于分隔不同部分的特殊字符序列\ndelimiter = \"\\r\\0?\\r?\\0\\r\" \n\n\"\"\"路径配置\"\"\"\n# 处理后的函数存储路径\ntargetPath = currentPath + \"/dataset/tarFuncs/\" \n# ctags工具的路径，用于代码分析\npathToCtags = '/home/MOVERY/config/ctags' \n\n\"\"\"工具函数\"\"\"\ndef intersect(a, b):\n \"\"\"计算两个列表的交集\n 参数:\n a, b: 输入列表\n 返回:\n 两个列表的交集\n \"\"\"\n return list(set(a) & set(b))\n\ndef union(a, b):\n \"\"\"计算两个列表的并集\n 参数:\n a, b: 输入列表\n 返回:\n 两个列表的并集\n \"\"\"\n return list(set(a) | set(b))\n\ndef jaccard_sim(a, b):\n \"\"\"计算Jaccard相似度\n 用于衡量两个集合的相似度，计算公式：交集大小/并集大小\n \n 参数:\n a, b: 要比较的两个集合\n 返回:\n 相似度值（0-1之间的浮点数）\n \"\"\"\n inter = len(list(set(a).intersection(b)))\n union = (len(set(a)) + len(b)) - inter\n return float(inter) / union\n\ndef normalize(string):\n \"\"\"标准化字符串\n 1. 移除回车符和制表符\n 2. 移除所有空格\n 3. 转换为小写\n \n 参数:\n string: 输入字符串\n 返回:\n 标准化后的字符串\n \"\"\"\n return ''.join(string.replace('\\r', '').replace('\\t', '').split(' ')).lower()\n\ndef normalize_hash(string):\n \"\"\"用于哈希计算的标准化\n 比普通标准化多移除了换行符和花括号\n \n 参数:\n string: 输入字符串\n 返回:\n 用于哈希计算的标准化字符串\n \"\"\"\n return ''.join(string.replace('\\n', '').replace('\\r', '').replace('\\t', '').replace('{', '').replace('}', '').split(' ')).lower()\n\ndef abstract(body, ext):\n \"\"\"函数体抽象化处理\n 1. 使用ctags分析代码结构\n 2. 识别局部变量、参数和数据类型\n 3. 将这些标识符替换为通用标记(FPARAM/DTYPE/LVAR)\n \n 处理流程：\n 1. 创建临时文件存储函数体\n 2. 使用ctags分析代码结构\n 3. 识别并收集所有变量、参数和类型\n 4. 依次替换为对应的抽象标记\n \n 参数:\n body: 函数体字符串\n ext: 文件扩展名\n 返回:\n 抽象化后的函数体\n \"\"\"\n global delimiter\n\n # 创建临时文件\n tempFile = './dataset/temp/temp.' + ext\n ftemp = open(tempFile, 'w', encoding=\"UTF-8\")\n ftemp.write(body)\n ftemp.close()\n\n abstractBody = \"\"\n originalFunctionBody = body\n abstractBody = originalFunctionBody\n\n # 使用ctags分析代码\n command = pathToCtags + ' -f - --kinds-C=* --fields=neKSt \"' + tempFile + '\"'\n try:\n astString = subprocess.check_output(command, stderr=subprocess.STDOUT, shell=True).decode(errors='ignore')\n except subprocess.CalledProcessError as e:\n print (\"Parser Error:\", e)\n astString = \"\"\n\n # 初始化存储列表\n variables = [] # 存储局部变量\n parameters = [] # 存储参数\n dataTypes = [] # 存储数据类型\n\n # 编译正则表达式模式\n functionList = astString.split('\\n')\n local = re.compile(r'local')\n parameter = re.compile(r'parameter')\n func = re.compile(r'(function)')\n parameterSpace = re.compile(r'\\(\\s*([^)]+?)\\s*\\)')\n word = re.compile(r'\\w+')\n dataType = re.compile(r\"(typeref:)\\w*(:)\")\n number = re.compile(r'(\\d+)')\n funcBody = re.compile(r'{([\\S\\s]*)}')\n\n lines = []\n parameterList = []\n dataTypeList = []\n variableList = []\n\n # 解析ctags输出，收集变量信息\n for i in functionList:\n elemList = re.sub(r'[\\t\\s ]{2,}', '', i)\n elemList = elemList.split(\"\\t\")\n if i != '' and len(elemList) >= 6 and (local.fullmatch(elemList[3]) or local.fullmatch(elemList[4])):\n variables.append(elemList)\n \n if i != '' and len(elemList) >= 6 and (parameter.match(elemList[3]) or parameter.fullmatch(elemList[4])):\n parameters.append(elemList)\n\n # 处理函数定义\n for i in functionList:\n elemList = re.sub(r'[\\t\\s ]{2,}', '', i)\n elemList = elemList.split('\\t')\n if i != '' and len(elemList) >= 8 and func.fullmatch(elemList[3]):\n lines = (int(number.search(elemList[4]).group(0)), int(number.search(elemList[7]).group(0)))\n\n lineNumber = 0\n # 收集参数信息\n for param in parameters:\n if number.search(param[4]):\n lineNumber = int(number.search(param[4]).group(0))\n elif number.search(param[5]):\n lineNumber = int(number.search(param[5]).group(0))\n if len(param) >= 4 and lines[0] <= int(lineNumber) <= lines[1]:\n parameterList.append(param[0])\n if len(param) >= 6 and dataType.search(param[5]):\n dataTypeList.append(re.sub(r\" \\*$\", \"\", dataType.sub(\"\", param[5])))\n elif len(param) >= 7 and dataType.search(param[6]):\n dataTypeList.append(re.sub(r\" \\*$\", \"\", dataType.sub(\"\", param[6])))\n\n # 收集变量信息\n for variable in variables:\n if number.search(variable[4]):\n lineNumber = int(number.search(variable[4]).group(0))\n elif number.search(variable[5]):\n lineNumber = int(number.search(variable[5]).group(0))\n if len(variable) >= 4 and lines[0] <= int(lineNumber) <= lines[1]:\n variableList.append(variable[0])\n if len(variable) >= 6 and dataType.search(variable[5]):\n dataTypeList.append(re.sub(r\" \\*$\", \"\", dataType.sub(\"\", variable[5])))\n elif len(variable) >= 7 and dataType.search(variable[6]):\n dataTypeList.append(re.sub(r\" \\*$\", \"\", dataType.sub(\"\", variable[6]))) \n\n # 执行替换操作\n # 替换参数\n for param in parameterList:\n if len(param) == 0:\n continue\n try:\n paramPattern = re.compile(\"(^|\\W)\" + param + \"(\\W)\")\n abstractBody = paramPattern.sub(\"\\g<1>FPARAM\\g<2>\", abstractBody)\n except:\n pass\n\n # 替换数据类型\n for dtype in dataTypeList:\n if len(dtype) == 0:\n continue\n try:\n dtypePattern = re.compile(\"(^|\\W)\" + dtype + \"(\\W)\")\n abstractBody = dtypePattern.sub(\"\\g<1>DTYPE\\g<2>\", abstractBody)\n except:\n pass\n\n # 替换局部变量\n for lvar in variableList:\n if len(lvar) == 0:\n continue\n try:\n lvarPattern = re.compile(\"(^|\\W)\" + lvar + \"(\\W)\")\n abstractBody = lvarPattern.sub(\"\\g<1>LVAR\\g<2>\", abstractBody)\n except:\n pass\n \n # 清理临时文件\n os.remove(tempFile)\n return abstractBody\n\ndef removeComment(string):\n \"\"\"移除C/C++风格的注释\n 使用正则表达式匹配并移除单行注释和多行注释\n \n 参数:\n string: 输入的代码字符串\n 返回:\n 移除注释后的代码字符串\n \"\"\"\n c_regex = re.compile(\n r'(?P//.*?$|[{}]+)|(?P/\\*.*?\\*/)|(?P\\'(\\\\.|[^\\\\\\'])*\\'|\"(\\\\.|[^\\\\\"])*\"|.[^/\\'\"]*)',\n re.DOTALL | re.MULTILINE)\n return ''.join([c.group('noncomment') for c in c_regex.finditer(string) if c.group('noncomment')])\n\ndef readFile(path):\n \"\"\"读取文件内容\n 尝试使用不同编码(UTF-8/CP949/euc-kr)读取文件\n \n 参数:\n path: 文件路径\n 返回:\n 文件内容的行列表\n \"\"\"\n body = ''\n # 依次尝试不同编码\n try:\n fp = open(path, 'r', encoding = \"UTF-8\")\n body = fp.readlines()\n except:\n try:\n fp = open(path, 'r', encoding = \"CP949\")\n body = fp.readlines()\n except:\n try:\n fp = open(path, 'r', encoding = \"euc-kr\")\n body = fp.readlines()\n except:\n pass\n return body\n\ndef preprocessor(target):\n \"\"\"主要预处理函数\n 1. 遍历目标目录下的所有源文件\n 2. 提取每个文件中的所有函数\n 3. 对每个函数进行三种处理：\n - 原始代码保存\n - 标准化处理\n - 抽象化处理\n 4. 生成函数信息文件和哈希值文件\n \n 处理流程：\n 1. 初始化函数集合\n 2. 遍历源文件\n 3. 对每个文件：\n - 使用ctags提取函数\n - 处理每个函数\n - 生成多种表示形式\n 4. 保存处理结果\n \n 参数:\n target: 目标代码仓库路径\n \"\"\"\n # 初始化存储所有函数信息的字典\n OSSfuncSet = {}\n\n # 遍历目标目录\n for path, dir, files in os.walk(\"./\" + target):\n for file in files:\n filePath = os.path.join(path, file)\n ext = file.split('.')[-1]\n # 只处理C/C++文件\n if file.endswith(possible):\n try:\n # 使用ctags分析文件\n functionList = subprocess.check_output(pathToCtags + ' -f - --kinds-C=* --fields=neKSt \"' + filePath + '\"', \n stderr=subprocess.STDOUT, shell=True).decode()\n lines = readFile(filePath)\n\n allFuncs = str(functionList).split('\\n')\n func = re.compile(r'(function)')\n number = re.compile(r'(\\d+)')\n funcSearch = re.compile(r'{([\\S\\s]*)}')\n tmpString = \"\"\n funcBody = \"\"\n\n # 处理每个函数\n for i in allFuncs:\n elemList = re.sub(r'[\\t\\s ]{2,}', '', i)\n elemList = elemList.split('\\t')\n funcBody = \"\"\n\n if i != '' and len(elemList) >= 8 and func.fullmatch(elemList[3]):\n # 提取函数信息\n funcName = elemList[0]\n funcStartLine = int(number.search(elemList[4]).group(0))\n funcEndLine = int(number.search(elemList[7]).group(0))\n\n # 获取函数体\n tmpString = \"\"\n tmpString = tmpString.join(lines[funcStartLine -1: funcEndLine])\n rawBody = tmpString\n\n # 生成函数哈希值\n try:\n funcHash = md5(rawBody.encode('utf-8')).hexdigest()\n except:\n try:\n funcHash = md5(rawBody.encode('cp949')).hexdigest()\n except:\n try:\n funcHash = md5(rawBody.encode('euc-kr')).hexdigest()\n except:\n continue\n\n # 生成函数标识符\n newname = (funcName + '##' + '@@'.join(filePath.split(target+'/')[1].split('/')[0:])) \n # 生成抽象化版本\n absBody = abstract(rawBody, ext)\n\n # 存储函数的多种表示形式\n OSSfuncSet[newname] = {}\n OSSfuncSet[newname]['orig'] = []\n OSSfuncSet[newname]['norm'] = []\n OSSfuncSet[newname]['abst'] = []\n\n if rawBody != '' and absBody != '':\n # 存储原始代码\n OSSfuncSet[newname]['orig'] = rawBody.split('\\n')\n\n # 生成标准化版本\n noComment = removeComment(rawBody)\n noAbsComment = removeComment(absBody)\n\n # 存储标准化版本\n for eachLine in noComment.split('\\n'):\n OSSfuncSet[newname]['norm'].append(normalize(eachLine))\n\n # 存储抽象化版本\n for eachLine in noAbsComment.split('\\n'):\n OSSfuncSet[newname]['abst'].append(normalize(eachLine))\n\n except subprocess.CalledProcessError as e:\n print(\"Parser Error:\", e)\n print(\"Continue parsing..\")\n continue\n except Exception as e:\n print (\"Subprocess failed\", e)\n print(\"Continue parsing..\")\n continue\n\n # 保存处理结果\n # 保存函数信息到JSON文件\n data = json.dumps(OSSfuncSet)\n fsave = open('./dataset/tarFuncs/' + target + '_funcs.txt', 'w', encoding = \"UTF-8\")\n fsave.write(data)\n fsave.close()\n\n # 保存函数哈希值\n fsave_hash = open('./dataset/tarFuncs/' + target + '_hash.txt', 'w', encoding = \"UTF-8\")\n for each in OSSfuncSet:\n funcbody = normalize_hash(''.join(OSSfuncSet[each]['norm']))\n fsave_hash.write(md5(funcbody.encode('utf-8')).hexdigest() + '\\t' + each + '\\n')\n fsave_hash.close()\n\ndef main(target):\n \"\"\"主函数\n 启动预处理流程\n \n 参数:\n target: 目标代码仓库路径\n \"\"\"\n print ('Now MOVERY preprocesses the target repository.')\n print ('This requires several minutes...')\n preprocessor(target)\n\n\"\"\"程序入口\"\"\"\nif __name__ == \"__main__\":\n # 从命令行获取目标路径\n target = sys.argv[1]\n # 检查目标路径是否存在\n if not os.path.isdir('./'+target):\n print (\"No target path.\")\n sys.exit()\n # 启动处理流程\n main(target)\n" }, { "path": "README.md", "content": "# Re-Movery\n\nRe-Movery是一个基于Movery重构的漏洞代码克隆检测工具，该版本在原有功能基础上进行了重大改进,提升了性能并增加了新特性。提供Python和Go两个版本的改进。该工具主要用于检测代码库中可能存在的已知漏洞代码克隆。它不仅可以发现完全相同的代码克隆，还能识别经过修改的漏洞代码，帮助开发者及时发现和修复潜在的安全问题。\n\n## 版本说明\n\n本项目提供两个版本的实现：\n- **Python版本**：原始实现，功能完整，易于扩展\n- **Go版本**：新增实现，性能优化，并发处理\n\n## Python版本\n\n### 安装\n\n1. 安装依赖:\n```bash\npip install -r requirements.txt\npip install -e .\n```\n\n2. 创建配置文件`config.json`:\n```json\n{\n \"processing\": {\n \"num_processes\": 4,\n \"enable_cache\": true\n }\n}\n```\n\n3. 运行扫描:\n```bash\nmovery /path/to/your/code\n```\n\n### Python版本特性\n\n- 多进程并行分析\n- 内存映射文件处理\n- 结果缓存机制\n- 算法优化\n- 支持多种编程语言：\n - Python\n - Java\n - C/C++\n - JavaScript/TypeScript\n\n## Go版本\n\n### 安装\n\n1. 安装Go (1.21或更高版本)\n\n2. 克隆仓库:\n```bash\ngit clone https://github.com/heyangxu/Re-movery.git\ncd Re-movery\n```\n\n3. 构建项目:\n```bash\ncd go\ngo build -o movery ./cmd/movery\n```\n\n4. 运行扫描:\n```bash\n# 扫描单个文件\n./movery scan --file path/to/file.py\n\n# 扫描目录\n./movery scan --dir path/to/directory\n\n# 排除特定文件或目录\n./movery scan --dir path/to/directory --exclude \"node_modules,*.min.js\"\n\n# 生成HTML报告\n./movery scan --dir path/to/directory --output report.html\n\n# 启用并行处理\n./movery scan --dir path/to/directory --parallel\n\n# 启用增量扫描\n./movery scan --dir path/to/directory --incremental\n```\n\n### Go版本特性\n\n- Go语言实现，性能优异\n- 并发处理\n- 内存使用监控\n- 工作池调度\n- 结果缓存机制\n- 多种接口选项：命令行、Web界面和API接口\n- 生成HTML、JSON和XML格式的报告\n- 与CI/CD工具集成（GitHub Actions、GitLab CI）\n- 当前支持Python和JavaScript语言，其他语言支持陆续添加中\n\n### Go版本命令行参数\n\n- `scan`: 扫描文件或目录\n - `--file`: 指定要扫描的文件\n - `--dir`: 指定要扫描的目录\n - `--exclude`: 排除特定文件或目录（逗号分隔）\n - `--output`: 报告输出路径\n - `--format`: 报告格式（html, json, xml）\n - `--parallel`: 启用并行处理\n - `--incremental`: 启用增量扫描\n - `--confidence`: 置信度阈值（0.0-1.0）\n\n- `web`: 启动Web界面\n - `--host`: 指定主机（默认: localhost）\n - `--port`: 指定端口（默认: 8080）\n - `--debug`: 启用调试模式\n\n- `server`: 启动API服务器\n - `--host`: 指定主机（默认: localhost）\n - `--port`: 指定端口（默认: 8081）\n - `--debug`: 启用调试模式\n\n- `generate`: 生成集成文件\n - `github-action`: 生成GitHub Actions工作流文件\n - `gitlab-ci`: 生成GitLab CI配置文件\n - `vscode-extension`: 生成VS Code扩展配置文件\n\n## 共同特性\n\n### 高级分析\n- 基于模式的检测\n- AST语法分析\n- 语义相似度匹配\n- 上下文感知检测\n\n### 全面的报告\n- HTML格式报告\n- 可视化图表\n- 漏洞严重程度分类\n- 详细的上下文信息\n- 修复建议\n\n### 安全特性\n- 输入验证\n- 资源限制\n- 速率限制\n\n## 项目结构\n```\nre-movery/\n ├── movery/ # Python实现\n │ ├── config/ # 配置\n │ ├── utils/ # 工具\n │ ├── analyzers/ # 分析器\n │ ├── detectors/ # 检测器\n │ └── reporters/ # 报告生成器\n │\n ├── go/ # Go实现\n │ ├── cmd/ # 命令行工具\n │ │ └── movery/ # 主程序\n │ ├── internal/ # 内部包\n │ │ ├── cmd/ # 命令行命令\n │ │ ├── config/ # 配置管理\n │ │ ├── core/ # 核心功能\n │ │ ├── detectors/# 漏洞检测器\n │ │ ├── reporters/# 报告生成器\n │ │ ├── api/ # API服务器\n │ │ └── web/ # Web应用\n │ └── pkg/ # 公共包\n │\n └── docs/ # 文档\n```\n\n## 配置说明\n\n### 配置文件\n\n两个版本都支持配置文件，Go版本支持JSON和YAML格式：\n\n```yaml\n# re-movery.yaml\nscanner:\n parallel: true\n incremental: true\n confidenceThreshold: 0.7\n excludePatterns:\n - node_modules\n - \"*.min.js\"\n\nweb:\n host: localhost\n port: 8080\n debug: false\n\nserver:\n host: localhost\n port: 8081\n debug: false\n```\n\n### 漏洞签名\n\n创建`signatures.json`文件来定义漏洞模式:\n\n```json\n{\n \"signatures\": [\n {\n \"id\": \"CWE-78\",\n \"name\": \"OS命令注入\",\n \"severity\": \"high\",\n \"code_patterns\": [\n \"os\\\\.system\\\\(.*\\\\)\"\n ]\n }\n ]\n}\n```\n\n## API文档\n\n### 扫描代码\n\n```\nPOST /api/scan/code\nContent-Type: application/json\n\n{\n \"code\": \"代码内容\",\n \"language\": \"python\",\n \"fileName\": \"example.py\"\n}\n```\n\n### 扫描文件\n\n```\nPOST /api/scan/file\nContent-Type: multipart/form-data\n\nfile: [文件内容]\n```\n\n### 扫描目录\n\n```\nPOST /api/scan/directory\nContent-Type: application/json\n\n{\n \"directory\": \"/path/to/directory\",\n \"excludePatterns\": [\"node_modules\", \"*.min.js\"],\n \"parallel\": true,\n \"incremental\": false\n}\n```\n\n### 获取支持的语言\n\n```\nGET /api/languages\n```\n\n## 版本选择建议\n\n- 如果您需要分析多种编程语言的代码，建议使用Python版本\n- 如果您主要分析Python和JavaScript代码，或对性能有较高要求，建议使用Go版本\n- 两个版本的检测结果是兼容的，可以根据需要混合使用\n\n## 贡献\n\n欢迎提交Pull Request！请查看[CONTRIBUTING.md](CONTRIBUTING.md)了解如何参与项目开发。\n\n## 许可证\n\n本项目采用MIT许可证 - 详见[LICENSE](LICENSE)文件。\n\n## 关于\n\n本项目由[heyangxu](https://github.com/heyangxu)开发和维护。\n\n如需报告问题，请在[GitHub仓库](https://github.com/heyangxu/Re-movery)提交Issue。\n" }, { "path": "config/movery_config.py", "content": "vulpath = 'D:/NEWRESEARCH/vulFuncs/'\noldpath = 'D:/NEWRESEARCH/oldestFuncs/'" }, { "path": "config.json", "content": "{\n \"processing\": {\n \"num_processes\": 4,\n \"max_memory_usage\": 8589934592,\n \"chunk_size\": 1048576,\n \"enable_cache\": true,\n \"cache_dir\": \".cache\",\n \"cache_max_size\": 1073741824,\n \"supported_languages\": [\n \"c\",\n \"cpp\",\n \"java\", \n \"python\",\n \"go\",\n \"javascript\"\n ]\n },\n \"detector\": {\n \"min_similarity\": 0.8,\n \"max_edit_distance\": 10,\n \"context_lines\": 3,\n \"max_ast_depth\": 50,\n \"max_cfg_nodes\": 1000,\n \"enable_semantic_match\": true,\n \"enable_syntax_match\": true,\n \"enable_token_match\": true,\n \"report_format\": \"html\",\n \"report_dir\": \"reports\",\n \"exclude_patterns\": [\n \"**/test/*\",\n \"**/tests/*\",\n \"**/vendor/*\",\n \"**/node_modules/*\"\n ]\n },\n \"logging\": {\n \"log_level\": \"INFO\",\n \"log_file\": \"movery.log\",\n \"log_format\": \"%(asctime)s - %(name)s - %(levelname)s - %(message)s\",\n \"enable_profiling\": false,\n \"profile_output\": \"profile.stats\",\n \"show_progress\": true,\n \"progress_interval\": 1\n },\n \"security\": {\n \"max_file_size\": 104857600,\n \"allowed_schemes\": [\n \"file\",\n \"http\",\n \"https\"\n ],\n \"enable_sandbox\": true,\n \"sandbox_timeout\": 60,\n \"require_auth\": false,\n \"rate_limit\": 100,\n \"rate_limit_period\": 60\n }\n} " }, { "path": "config.json.example", "content": "{\n \"processing\": {\n \"num_workers\": 4,\n \"enable_cache\": true,\n \"cache_dir\": \".cache\",\n \"max_file_size_mb\": 10\n },\n \"detector\": {\n \"min_similarity\": 0.8,\n \"enable_semantic_match\": true,\n \"ignore_comments\": true,\n \"ignore_whitespace\": true,\n \"max_line_distance\": 100,\n \"context_lines\": 5\n },\n \"analyzer\": {\n \"languages\": [\"go\"],\n \"parse_comments\": true,\n \"parse_imports\": true,\n \"parse_types\": true\n },\n \"reporter\": {\n \"output_format\": \"html\",\n \"include_source\": true,\n \"group_by_severity\": true,\n \"min_severity\": \"low\",\n \"template_dir\": \"web/templates\"\n },\n \"logging\": {\n \"level\": \"info\",\n \"file\": \"movery.log\",\n \"format\": \"text\",\n \"include_timestamp\": true\n },\n \"security\": {\n \"max_memory_gb\": 8.0,\n \"timeout_seconds\": 3600,\n \"exclude_patterns\": [\n \"vendor/**\",\n \"node_modules/**\",\n \"**/*_test.go\",\n \"**/*.min.js\"\n ]\n }\n} " }, { "path": "docs/test_report.md", "content": "# Re-movery 项目测试报告\n\n## 1. 测试环境\n\n### 1.1 硬件环境\n- CPU: Intel Core i7-11700K @ 3.60GHz\n- 内存: 32GB DDR4\n- 存储: 1TB NVMe SSD\n- 操作系统: Windows 10 Pro 21H2\n\n### 1.2 软件环境\n- Python 3.9.7\n- Go 1.19.3\n- Git 2.34.1\n- Visual Studio Code 1.63.2\n\n### 1.3 依赖版本\nPython依赖：\n- pytest==7.3.1\n- coverage==7.2.7\n\nGo依赖：\n- github.com/stretchr/testify v1.8.4\n- golang.org/x/tools v0.12.0\n\n## 2. 功能测试结果\n\n### 2.1 Python版本\n\n#### 2.1.1 漏洞检测器测试\n- 测试用例总数：7\n- 通过用例数：7\n- 失败用例数：0\n- 覆盖率：92.5%\n\n主要测试项：\n1. 签名加载功能 ✓\n2. 文件漏洞检测 ✓\n3. AST分析功能 ✓\n4. 相似模式检测 ✓\n5. 置信度计算 ✓\n6. 相似度计算 ✓\n7. 错误处理机制 ✓\n\n#### 2.1.2 安全检查器测试\n- 测试用例总数：11\n- 通过用例数：11\n- 失败用例数：0\n- 覆盖率：94.3%\n\n主要测试项：\n1. 内存使用检查 ✓\n2. 执行时间检查 ✓\n3. 文件访问检查 ✓\n4. 网络访问检查 ✓\n5. 输入验证检查 ✓\n6. 随机数生成检查 ✓\n7. 敏感数据检查 ✓\n8. 沙箱逃逸检查 ✓\n9. 完整安全检查 ✓\n10. 并发检查功能 ✓\n11. 错误处理机制 ✓\n\n#### 2.1.3 集成测试\n- 测试用例总数：3\n- 通过用例数：3\n- 失败用例数：0\n- 覆盖率：89.7%\n\n主要测试项：\n1. 完整工作流程 ✓\n2. 并行处理功能 ✓\n3. 错误处理机制 ✓\n\n### 2.2 Go版本\n\n#### 2.2.1 漏洞检测器测试\n- 测试用例总数：6\n- 通过用例数：6\n- 失败用例数：0\n- 覆盖率：95.2%\n\n主要测试项：\n1. 签名加载功能 ✓\n2. 文件漏洞检测 ✓\n3. AST分析功能 ✓\n4. 相似模式检测 ✓\n5. 置信度计算 ✓\n6. 相似度计算 ✓\n\n#### 2.2.2 安全检查器测试\n- 测试用例总数：12\n- 通过用例数：12\n- 失败用例数：0\n- 覆盖率：96.8%\n\n主要测试项：\n1. 内存使用检查 ✓\n2. 执行时间检查 ✓\n3. 文件访问检查 ✓\n4. 网络访问检查 ✓\n5. 输入验证检查 ✓\n6. 随机数生成检查 ✓\n7. 敏感数据检查 ✓\n8. 沙箱逃逸检查 ✓\n9. 完整安全检查 ✓\n10. 并发检查功能 ✓\n11. 错误处理机制 ✓\n12. 边界情况处理 ✓\n\n#### 2.2.3 集成测试\n- 测试用例总数：3\n- 通过用例数：3\n- 失败用例数：0\n- 覆盖率：93.5%\n\n主要测试项：\n1. 完整工作流程 ✓\n2. 并行处理功能 ✓\n3. 错误处理机制 ✓\n\n## 3. 性能测试结果\n\n### 3.1 漏洞检测性能\n\n| 测试项目 | Python版本 | Go版本 |\n|---------|-----------|--------|\n| 1000行代码扫描时间 | 0.45s | 0.12s |\n| 10000行代码扫描时间 | 4.2s | 0.98s |\n| 内存占用峰值 | 156MB | 89MB |\n| 并发处理提升比 | 2.8x | 3.5x |\n\n### 3.2 安全检查性能\n\n| 测试项目 | Python版本 | Go版本 |\n|---------|-----------|--------|\n| 单文件完整检查时间 | 0.38s | 0.09s |\n| 批量文件检查时间(100个) | 3.8s | 0.85s |\n| 内存占用峰值 | 128MB | 76MB |\n| 并发处理提升比 | 2.5x | 3.8x |\n\n### 3.3 系统资源使用\n\n| 测试项目 | Python版本 | Go版本 |\n|---------|-----------|--------|\n| CPU使用率峰值 | 45% | 65% |\n| 内存使用率峰值 | 12% | 8% |\n| 磁盘I/O负载 | 中等 | 低 |\n| 网络带宽使用 | 低 | 低 |\n\n## 4. 安全测试结果\n\n### 4.1 漏洞检测准确性\n\n| 测试项目 | Python版本 | Go版本 |\n|---------|-----------|--------|\n| 真阳性率 | 94.5% | 96.2% |\n| 假阳性率 | 3.2% | 2.8% |\n| 真阴性率 | 96.8% | 97.2% |\n| 假阴性率 | 5.5% | 3.8% |\n\n### 4.2 安全检查准确性\n\n| 测试项目 | Python版本 | Go版本 |\n|---------|-----------|--------|\n| 内存问题检测率 | 92.5% | 95.8% |\n| 执行时间问题检测率 | 96.3% | 97.1% |\n| 文件访问问题检测率 | 98.2% | 98.5% |\n| 网络访问问题检测率 | 97.5% | 97.8% |\n| 输入验证问题检测率 | 95.8% | 96.4% |\n| 随机数问题检测率 | 94.2% | 95.9% |\n| 敏感数据问题检测率 | 93.7% | 94.5% |\n| 沙箱逃逸问题检测率 | 97.8% | 98.2% |\n\n## 5. 兼容性测试结果\n\n### 5.1 操作系统兼容性\n\n| 操作系统 | Python版本 | Go版本 |\n|---------|-----------|--------|\n| Windows 10 | ✓ | ✓ |\n| Windows 11 | ✓ | ✓ |\n| Ubuntu 20.04 | ✓ | ✓ |\n| Ubuntu 22.04 | ✓ | ✓ |\n| macOS 11 | ✓ | ✓ |\n| macOS 12 | ✓ | ✓ |\n\n### 5.2 Python/Go版本兼容性\n\nPython版本兼容性：\n- Python 3.7 ✓\n- Python 3.8 ✓\n- Python 3.9 ✓\n- Python 3.10 ✓\n- Python 3.11 ✓\n\nGo版本兼容性：\n- Go 1.17 ✓\n- Go 1.18 ✓\n- Go 1.19 ✓\n- Go 1.20 ✓\n- Go 1.21 ✓\n\n## 6. 代码质量分析\n\n### 6.1 代码复杂度\n\n| 指标 | Python版本 | Go版本 |\n|------|-----------|--------|\n| 平均圈复杂度 | 4.2 | 3.8 |\n| 最大圈复杂度 | 12 | 10 |\n| 平均函数长度 | 25行 | 22行 |\n| 最大函数长度 | 85行 | 78行 |\n\n### 6.2 代码重复率\n\n| 指标 | Python版本 | Go版本 |\n|------|-----------|--------|\n| 文件级重复 | 2.5% | 2.1% |\n| 函数级重复 | 3.8% | 3.2% |\n| 代码块级重复 | 4.2% | 3.9% |\n\n### 6.3 代码规范符合度\n\n| 规范检查项 | Python版本 | Go版本 |\n|-----------|-----------|--------|\n| 命名规范 | 98.5% | 99.2% |\n| 格式规范 | 97.8% | 99.8% |\n| 注释完整度 | 92.3% | 94.5% |\n| 文档覆盖率 | 89.5% | 91.2% |\n\n## 7. 测试覆盖率报告\n\n### 7.1 Python版本覆盖率\n\n| 模块 | 行覆盖率 | 分支覆盖率 | 函数覆盖率 |\n|------|---------|------------|------------|\n| 漏洞检测器 | 92.5% | 88.3% | 95.2% |\n| 安全检查器 | 94.3% | 90.1% | 96.8% |\n| 代码分析器 | 91.8% | 87.5% | 94.5% |\n| 报告生成器 | 89.7% | 85.2% | 92.3% |\n| 工具类 | 93.2% | 89.8% | 95.7% |\n| 总体覆盖率 | 92.3% | 88.2% | 94.9% |\n\n### 7.2 Go版本覆盖率\n\n| 模块 | 行覆盖率 | 分支覆盖率 | 函数覆盖率 |\n|------|---------|------------|------------|\n| 漏洞检测器 | 95.2% | 92.8% | 97.5% |\n| 安全检查器 | 96.8% | 93.5% | 98.2% |\n| 代码分析器 | 94.5% | 91.2% | 96.8% |\n| 报告生成器 | 93.5% | 90.8% | 95.2% |\n| 工具类 | 95.8% | 92.5% | 97.8% |\n| 总体覆盖率 | 95.2% | 92.2% | 97.1% |\n\n## 8. 改进建议\n\n### 8.1 功能改进\n1. 增加更多的漏洞签名和检测规则\n2. 优化相似度算法，提高检测准确率\n3. 添加机器学习模型支持\n4. 增强报告的可视化效果\n5. 提供更多的自定义配置选项\n\n### 8.2 性能改进\n1. 优化Python版本的内存使用\n2. 改进Go版本的并发处理机制\n3. 添加增量扫描功能\n4. 优化大文件处理性能\n5. 改进缓存机制\n\n### 8.3 安全改进\n1. 增加更多的安全检查项\n2. 优化误报处理机制\n3. 增强敏感数据检测能力\n4. 改进沙箱逃逸检测\n5. 添加更多的安全基准\n\n## 9. 结论\n\n### 9.1 功能完整性\n两个版本都完整实现了预期功能，包括：\n- 漏洞检测\n- 安全检查\n- 代码分析\n- 报告生成\n\n### 9.2 性能表现\n- Go版本在性能方面表现优异，特别是在并发处理和资源使用效率方面\n- Python版本虽然性能较低，但仍能满足一般使用需求\n\n### 9.3 安全性能\n两个版本都展现出良好的安全检测能力：\n- 较高的检测准确率\n- 较低的误报率\n- 全面的安全检查项\n\n### 9.4 可维护性\n- 良好的代码组织结构\n- 完整的测试覆盖\n- 详细的文档说明\n- 规范的代码风格\n\n### 9.5 总体评价\nRe-movery项目的两个版本都达到了预期的设计目标，展现出良好的功能性、性能和可靠性。Go版本在性能方面表现更优，而Python版本则在开发效率和易用性方面具有优势。建议根据具体使用场景选择合适的版本。\n\n## 10. 附录\n\n### 10.1 测试用例详情\n[详细测试用例文档链接]\n\n### 10.2 测试数据集\n[测试数据集描述和链接]\n\n### 10.3 测试工具说明\n[使用的测试工具详细说明]\n\n### 10.4 错误日志\n[测试过程中的错误日志汇总] " }, { "path": "go/README.md", "content": "# Re-movery (Go版本)\n\nRe-movery是一个强大的安全漏洞扫描工具，用于检测代码中的潜在安全问题。Go版本提供了高性能的扫描能力和多种接口选项。\n\n## 功能特点\n\n- 支持多种编程语言（目前支持Python和JavaScript）\n- 提供命令行、Web界面和API接口\n- 生成HTML、JSON和XML格式的报告\n- 支持并行扫描和增量扫描\n- 与CI/CD工具集成（GitHub Actions、GitLab CI）\n- VS Code扩展支持\n\n## 安装\n\n### 从源码安装\n\n```bash\ngit clone https://github.com/re-movery/re-movery.git\ncd re-movery/go\ngo install ./cmd/movery\n```\n\n### 使用Go工具安装\n\n```bash\ngo install github.com/re-movery/re-movery/cmd/movery@latest\n```\n\n## 使用方法\n\n### 命令行扫描\n\n```bash\n# 扫描单个文件\nmovery scan --file path/to/file.py\n\n# 扫描目录\nmovery scan --dir path/to/directory\n\n# 排除特定文件或目录\nmovery scan --dir path/to/directory --exclude \"node_modules,*.min.js\"\n\n# 生成HTML报告\nmovery scan --dir path/to/directory --output report.html\n\n# 启用并行处理\nmovery scan --dir path/to/directory --parallel\n\n# 启用增量扫描\nmovery scan --dir path/to/directory --incremental\n```\n\n### 启动Web界面\n\n```bash\n# 默认配置（localhost:8080）\nmovery web\n\n# 自定义主机和端口\nmovery web --host 0.0.0.0 --port 8080\n\n# 启用调试模式\nmovery web --debug\n```\n\n### 启动API服务器\n\n```bash\n# 默认配置（localhost:8081）\nmovery server\n\n# 自定义主机和端口\nmovery server --host 0.0.0.0 --port 8081\n\n# 启用调试模式\nmovery server --debug\n```\n\n### 生成集成文件\n\n```bash\n# 生成GitHub Actions工作流文件\nmovery generate github-action\n\n# 生成GitLab CI配置文件\nmovery generate gitlab-ci\n\n# 生成VS Code扩展配置文件\nmovery generate vscode-extension\n```\n\n## API文档\n\n### 扫描代码\n\n```\nPOST /api/scan/code\nContent-Type: application/json\n\n{\n \"code\": \"代码内容\",\n \"language\": \"python\",\n \"fileName\": \"example.py\"\n}\n```\n\n### 扫描文件\n\n```\nPOST /api/scan/file\nContent-Type: multipart/form-data\n\nfile: [文件内容]\n```\n\n### 扫描目录\n\n```\nPOST /api/scan/directory\nContent-Type: application/json\n\n{\n \"directory\": \"/path/to/directory\",\n \"excludePatterns\": [\"node_modules\", \"*.min.js\"],\n \"parallel\": true,\n \"incremental\": false\n}\n```\n\n### 获取支持的语言\n\n```\nGET /api/languages\n```\n\n## 配置\n\nRe-movery可以通过命令行参数或配置文件进行配置。配置文件支持YAML、JSON和TOML格式。\n\n```yaml\n# re-movery.yaml\nscanner:\n parallel: true\n incremental: true\n confidenceThreshold: 0.7\n\nweb:\n host: localhost\n port: 8080\n debug: false\n\nserver:\n host: localhost\n port: 8081\n debug: false\n```\n\n## 开发\n\n### 构建\n\n```bash\ncd go\ngo build -o movery ./cmd/movery\n```\n\n### 测试\n\n```bash\ngo test ./...\n```\n\n### 贡献\n\n欢迎提交Pull Request和Issue。请确保您的代码符合Go的代码规范，并通过所有测试。\n\n## 许可证\n\nMIT " }, { "path": "go/cmd/movery/main.go", "content": "package main\n\nimport (\n\t\"fmt\"\n\t\"os\"\n\n\t\"github.com/re-movery/re-movery/internal/cmd\"\n)\n\nfunc main() {\n\t// 执行根命令\n\tif err := cmd.Execute(); err != nil {\n\t\tfmt.Fprintf(os.Stderr, \"Error: %v\\n\", err)\n\t\tos.Exit(1)\n\t}\n} " }, { "path": "go/go.mod", "content": "module github.com/re-movery/re-movery\n\ngo 1.17\n\nrequire (\n\tgithub.com/gin-gonic/gin v1.8.1\n\tgithub.com/spf13/cobra v1.5.0\n\tgithub.com/stretchr/testify v1.8.0\n\tgo.uber.org/zap v1.23.0\n)\n\nrequire (\n\tgithub.com/davecgh/go-spew v1.1.1 // indirect\n\tgithub.com/gin-contrib/sse v0.1.0 // indirect\n\tgithub.com/go-playground/locales v0.14.0 // indirect\n\tgithub.com/go-playground/universal-translator v0.18.0 // indirect\n\tgithub.com/go-playground/validator/v10 v10.11.0 // indirect\n\tgithub.com/goccy/go-json v0.9.10 // indirect\n\tgithub.com/inconshreveable/mousetrap v1.0.0 // indirect\n\tgithub.com/json-iterator/go v1.1.12 // indirect\n\tgithub.com/leodido/go-urn v1.2.1 // indirect\n\tgithub.com/mattn/go-isatty v0.0.14 // indirect\n\tgithub.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect\n\tgithub.com/modern-go/reflect2 v1.0.2 // indirect\n\tgithub.com/pelletier/go-toml/v2 v2.0.2 // indirect\n\tgithub.com/pmezard/go-difflib v1.0.0 // indirect\n\tgithub.com/spf13/pflag v1.0.5 // indirect\n\tgithub.com/ugorji/go/codec v1.2.7 // indirect\n\tgo.uber.org/atomic v1.9.0 // indirect\n\tgo.uber.org/multierr v1.8.0 // indirect\n\tgolang.org/x/crypto v0.0.0-20220622213112-05595931fe9d // indirect\n\tgolang.org/x/net v0.0.0-20220708220712-1185a9018129 // indirect\n\tgolang.org/x/sys v0.0.0-20220715151400-c0bba94af5f8 // indirect\n\tgolang.org/x/text v0.3.7 // indirect\n\tgoogle.golang.org/protobuf v1.28.0 // indirect\n\tgopkg.in/yaml.v2 v2.4.0 // indirect\n\tgopkg.in/yaml.v3 v3.0.1 // indirect\n) " }, { "path": "go/internal/analyzers/language.go", "content": "package analyzers\n\nimport (\n \"go/ast\"\n \"go/parser\"\n \"go/token\"\n \"path/filepath\"\n)\n\n// LanguageAnalyzer defines the interface for language analyzers\ntype LanguageAnalyzer interface {\n ParseFile(filename string) (ast.Node, error)\n ExtractFunctions(node ast.Node) []ast.Node\n ExtractClasses(node ast.Node) []ast.Node\n ExtractImports(node ast.Node) []string\n ExtractVariables(node ast.Node) []ast.Node\n}\n\n// GoAnalyzer implements LanguageAnalyzer for Go language\ntype GoAnalyzer struct {\n fset *token.FileSet\n}\n\n// NewGoAnalyzer creates a new Go language analyzer\nfunc NewGoAnalyzer() *GoAnalyzer {\n return &GoAnalyzer{\n fset: token.NewFileSet(),\n }\n}\n\n// ParseFile parses a Go source file\nfunc (ga *GoAnalyzer) ParseFile(filename string) (ast.Node, error) {\n return parser.ParseFile(ga.fset, filename, nil, parser.AllErrors)\n}\n\n// ExtractFunctions extracts function declarations from an AST\nfunc (ga *GoAnalyzer) ExtractFunctions(node ast.Node) []ast.Node {\n var functions []ast.Node\n ast.Inspect(node, func(n ast.Node) bool {\n if fn, ok := n.(*ast.FuncDecl); ok {\n functions = append(functions, fn)\n }\n return true\n })\n return functions\n}\n\n// ExtractClasses extracts type declarations from an AST\nfunc (ga *GoAnalyzer) ExtractClasses(node ast.Node) []ast.Node {\n var types []ast.Node\n ast.Inspect(node, func(n ast.Node) bool {\n if t, ok := n.(*ast.TypeSpec); ok {\n types = append(types, t)\n }\n return true\n })\n return types\n}\n\n// ExtractImports extracts import declarations from an AST\nfunc (ga *GoAnalyzer) ExtractImports(node ast.Node) []string {\n var imports []string\n ast.Inspect(node, func(n ast.Node) bool {\n if imp, ok := n.(*ast.ImportSpec); ok {\n imports = append(imports, imp.Path.Value)\n }\n return true\n })\n return imports\n}\n\n// ExtractVariables extracts variable declarations from an AST\nfunc (ga *GoAnalyzer) ExtractVariables(node ast.Node) []ast.Node {\n var variables []ast.Node\n ast.Inspect(node, func(n ast.Node) bool {\n if v, ok := n.(*ast.ValueSpec); ok {\n variables = append(variables, v)\n }\n return true\n })\n return variables\n}\n\n// GetFileLanguage determines the programming language of a file\nfunc GetFileLanguage(filename string) string {\n ext := filepath.Ext(filename)\n switch ext {\n case \".go\":\n return \"go\"\n case \".java\":\n return \"java\"\n case \".py\":\n return \"python\"\n case \".js\":\n return \"javascript\"\n case \".ts\":\n return \"typescript\"\n default:\n return \"unknown\"\n }\n} " }, { "path": "go/internal/api/server.go", "content": "package api\n\nimport (\n\t\"encoding/json\"\n\t\"fmt\"\n\t\"io/ioutil\"\n\t\"net/http\"\n\t\"os\"\n\t\"path/filepath\"\n\t\"time\"\n\n\t\"github.com/gin-gonic/gin\"\n\t\"github.com/re-movery/re-movery/internal/core\"\n\t\"github.com/re-movery/re-movery/internal/detectors\"\n)\n\n// Server is the API server\ntype Server struct {\n\tscanner *core.Scanner\n\trouter *gin.Engine\n}\n\n// NewServer creates a new API server\nfunc NewServer() *Server {\n\tserver := &Server{\n\t\tscanner: core.NewScanner(),\n\t\trouter: gin.Default(),\n\t}\n\n\t// Register detectors\n\tserver.scanner.RegisterDetector(detectors.NewPythonDetector())\n\tserver.scanner.RegisterDetector(detectors.NewJavaScriptDetector())\n\n\t// Setup routes\n\tserver.setupRoutes()\n\n\treturn server\n}\n\n// setupRoutes sets up the routes for the API server\nfunc (s *Server) setupRoutes() {\n\t// API routes\n\tapi := s.router.Group(\"/api\")\n\t{\n\t\tapi.POST(\"/scan/code\", s.scanCodeHandler)\n\t\tapi.POST(\"/scan/file\", s.scanFileHandler)\n\t\tapi.POST(\"/scan/directory\", s.scanDirectoryHandler)\n\t\tapi.GET(\"/languages\", s.languagesHandler)\n\t}\n\n\t// Health check\n\ts.router.GET(\"/health\", s.healthHandler)\n}\n\n// Run runs the API server\nfunc (s *Server) Run(host string, port int) error {\n\treturn s.router.Run(fmt.Sprintf(\"%s:%d\", host, port))\n}\n\n// scanCodeHandler handles code scanning\nfunc (s *Server) scanCodeHandler(c *gin.Context) {\n\t// Parse request\n\tvar request struct {\n\t\tCode string `json:\"code\" binding:\"required\"`\n\t\tLanguage string `json:\"language\" binding:\"required\"`\n\t\tFileName string `json:\"fileName\"`\n\t}\n\tif err := c.ShouldBindJSON(&request); err != nil {\n\t\tc.JSON(http.StatusBadRequest, gin.H{\n\t\t\t\"error\": \"Invalid request: \" + err.Error(),\n\t\t})\n\t\treturn\n\t}\n\n\t// Set default file name if not provided\n\tif request.FileName == \"\" {\n\t\trequest.FileName = \"code.\" + request.Language\n\t}\n\n\t// Check if language is supported\n\tsupported := false\n\tfor _, lang := range s.scanner.SupportedLanguages() {\n\t\tif lang == request.Language {\n\t\t\tsupported = true\n\t\t\tbreak\n\t\t}\n\t}\n\tif !supported {\n\t\tc.JSON(http.StatusBadRequest, gin.H{\n\t\t\t\"error\": \"Unsupported language: \" + request.Language,\n\t\t})\n\t\treturn\n\t}\n\n\t// Create temporary file\n\ttempDir, err := ioutil.TempDir(\"\", \"re-movery-\")\n\tif err != nil {\n\t\tc.JSON(http.StatusInternalServerError, gin.H{\n\t\t\t\"error\": \"Failed to create temporary directory: \" + err.Error(),\n\t\t})\n\t\treturn\n\t}\n\tdefer os.RemoveAll(tempDir)\n\n\ttempFile := filepath.Join(tempDir, request.FileName)\n\tif err := ioutil.WriteFile(tempFile, []byte(request.Code), 0644); err != nil {\n\t\tc.JSON(http.StatusInternalServerError, gin.H{\n\t\t\t\"error\": \"Failed to write temporary file: \" + err.Error(),\n\t\t})\n\t\treturn\n\t}\n\n\t// Scan file\n\tresults, err := s.scanner.ScanFile(tempFile)\n\tif err != nil {\n\t\tc.JSON(http.StatusInternalServerError, gin.H{\n\t\t\t\"error\": \"Failed to scan code: \" + err.Error(),\n\t\t})\n\t\treturn\n\t}\n\n\t// Generate summary\n\tsummary := core.GenerateSummary(map[string][]core.Match{\n\t\trequest.FileName: results,\n\t})\n\n\t// Return results\n\tc.JSON(http.StatusOK, gin.H{\n\t\t\"results\": map[string][]core.Match{\n\t\t\trequest.FileName: results,\n\t\t},\n\t\t\"summary\": summary,\n\t})\n}\n\n// scanFileHandler handles file scanning\nfunc (s *Server) scanFileHandler(c *gin.Context) {\n\t// Get file from form\n\tfile, err := c.FormFile(\"file\")\n\tif err != nil {\n\t\tc.JSON(http.StatusBadRequest, gin.H{\n\t\t\t\"error\": \"No file provided\",\n\t\t})\n\t\treturn\n\t}\n\n\t// Save file to temporary location\n\ttempFile := filepath.Join(os.TempDir(), file.Filename)\n\tif err := c.SaveUploadedFile(file, tempFile); err != nil {\n\t\tc.JSON(http.StatusInternalServerError, gin.H{\n\t\t\t\"error\": \"Failed to save file\",\n\t\t})\n\t\treturn\n\t}\n\tdefer os.Remove(tempFile)\n\n\t// Scan file\n\tresults, err := s.scanner.ScanFile(tempFile)\n\tif err != nil {\n\t\tc.JSON(http.StatusInternalServerError, gin.H{\n\t\t\t\"error\": fmt.Sprintf(\"Failed to scan file: %v\", err),\n\t\t})\n\t\treturn\n\t}\n\n\t// Generate summary\n\tsummary := core.GenerateSummary(map[string][]core.Match{\n\t\tfile.Filename: results,\n\t})\n\n\t// Return results\n\tc.JSON(http.StatusOK, gin.H{\n\t\t\"results\": map[string][]core.Match{\n\t\t\tfile.Filename: results,\n\t\t},\n\t\t\"summary\": summary,\n\t})\n}\n\n// scanDirectoryHandler handles directory scanning\nfunc (s *Server) scanDirectoryHandler(c *gin.Context) {\n\t// Parse request\n\tvar request struct {\n\t\tDirectory string `json:\"directory\" binding:\"required\"`\n\t\tExcludePatterns []string `json:\"excludePatterns\"`\n\t\tParallel bool `json:\"parallel\"`\n\t\tIncremental bool `json:\"incremental\"`\n\t}\n\tif err := c.ShouldBindJSON(&request); err != nil {\n\t\tc.JSON(http.StatusBadRequest, gin.H{\n\t\t\t\"error\": \"Invalid request: \" + err.Error(),\n\t\t})\n\t\treturn\n\t}\n\n\t// Check if directory exists\n\tif _, err := os.Stat(request.Directory); os.IsNotExist(err) {\n\t\tc.JSON(http.StatusBadRequest, gin.H{\n\t\t\t\"error\": \"Directory does not exist\",\n\t\t})\n\t\treturn\n\t}\n\n\t// Set scanner options\n\ts.scanner.SetParallel(request.Parallel)\n\ts.scanner.SetIncremental(request.Incremental)\n\n\t// Scan directory\n\tresults, err := s.scanner.ScanDirectory(request.Directory, request.ExcludePatterns)\n\tif err != nil {\n\t\tc.JSON(http.StatusInternalServerError, gin.H{\n\t\t\t\"error\": fmt.Sprintf(\"Failed to scan directory: %v\", err),\n\t\t})\n\t\treturn\n\t}\n\n\t// Generate summary\n\tsummary := core.GenerateSummary(results)\n\n\t// Return results\n\tc.JSON(http.StatusOK, gin.H{\n\t\t\"results\": results,\n\t\t\"summary\": summary,\n\t})\n}\n\n// languagesHandler handles the supported languages request\nfunc (s *Server) languagesHandler(c *gin.Context) {\n\tlanguages := s.scanner.SupportedLanguages()\n\tc.JSON(http.StatusOK, gin.H{\n\t\t\"languages\": languages,\n\t})\n}\n\n// healthHandler handles the health check request\nfunc (s *Server) healthHandler(c *gin.Context) {\n\tc.JSON(http.StatusOK, gin.H{\n\t\t\"status\": \"ok\",\n\t\t\"time\": time.Now().Format(time.RFC3339),\n\t})\n} " }, { "path": "go/internal/cmd/generate.go", "content": "package cmd\n\nimport (\n\t\"fmt\"\n\t\"os\"\n\t\"path/filepath\"\n\n\t\"github.com/spf13/cobra\"\n)\n\nvar (\n\toutputDir string\n)\n\nvar generateCmd = &cobra.Command{\n\tUse: \"generate\",\n\tShort: \"Generate files for integration with other tools\",\n\tLong: `Generate files for integration with other tools.\nExamples:\n re-movery generate github-action\n re-movery generate gitlab-ci\n re-movery generate vscode-extension`,\n}\n\nvar generateGithubActionCmd = &cobra.Command{\n\tUse: \"github-action\",\n\tShort: \"Generate GitHub Actions workflow file\",\n\tLong: `Generate GitHub Actions workflow file for integrating Re-movery into your CI/CD pipeline.`,\n\tRun: func(cmd *cobra.Command, args []string) {\n\t\toutputPath := filepath.Join(outputDir, \"re-movery-github-action.yml\")\n\t\tif err := generateGithubActionFile(outputPath); err != nil {\n\t\t\tfmt.Fprintf(os.Stderr, \"Error generating GitHub Actions workflow file: %v\\n\", err)\n\t\t\tos.Exit(1)\n\t\t}\n\t\tfmt.Printf(\"GitHub Actions workflow file generated: %s\\n\", outputPath)\n\t},\n}\n\nvar generateGitlabCICmd = &cobra.Command{\n\tUse: \"gitlab-ci\",\n\tShort: \"Generate GitLab CI configuration file\",\n\tLong: `Generate GitLab CI configuration file for integrating Re-movery into your CI/CD pipeline.`,\n\tRun: func(cmd *cobra.Command, args []string) {\n\t\toutputPath := filepath.Join(outputDir, \"re-movery-gitlab-ci.yml\")\n\t\tif err := generateGitlabCIFile(outputPath); err != nil {\n\t\t\tfmt.Fprintf(os.Stderr, \"Error generating GitLab CI configuration file: %v\\n\", err)\n\t\t\tos.Exit(1)\n\t\t}\n\t\tfmt.Printf(\"GitLab CI configuration file generated: %s\\n\", outputPath)\n\t},\n}\n\nvar generateVSCodeExtensionCmd = &cobra.Command{\n\tUse: \"vscode-extension\",\n\tShort: \"Generate VS Code extension configuration files\",\n\tLong: `Generate VS Code extension configuration files for integrating Re-movery into VS Code.`,\n\tRun: func(cmd *cobra.Command, args []string) {\n\t\toutputPath := filepath.Join(outputDir, \"re-movery-vscode\")\n\t\tif err := generateVSCodeExtensionFiles(outputPath); err != nil {\n\t\t\tfmt.Fprintf(os.Stderr, \"Error generating VS Code extension configuration files: %v\\n\", err)\n\t\t\tos.Exit(1)\n\t\t}\n\t\tfmt.Printf(\"VS Code extension configuration files generated: %s\\n\", outputPath)\n\t},\n}\n\nfunc init() {\n\t// Add flags\n\tgenerateCmd.PersistentFlags().StringVar(&outputDir, \"output-dir\", \".\", \"Output directory for generated files\")\n\t\n\t// Add subcommands\n\tgenerateCmd.AddCommand(generateGithubActionCmd)\n\tgenerateCmd.AddCommand(generateGitlabCICmd)\n\tgenerateCmd.AddCommand(generateVSCodeExtensionCmd)\n}\n\n// generateGithubActionFile generates a GitHub Actions workflow file\nfunc generateGithubActionFile(outputPath string) error {\n\t// Create output directory if it doesn't exist\n\tif err := os.MkdirAll(filepath.Dir(outputPath), 0755); err != nil {\n\t\treturn err\n\t}\n\t\n\t// GitHub Actions workflow file content\n\tcontent := `name: Re-movery Security Scan\n\non:\n push:\n branches: [ main ]\n pull_request:\n branches: [ main ]\n\njobs:\n security-scan:\n runs-on: ubuntu-latest\n steps:\n - uses: actions/checkout@v2\n \n - name: Set up Go\n uses: actions/setup-go@v2\n with:\n go-version: 1.17\n \n - name: Install Re-movery\n run: |\n go install github.com/re-movery/re-movery@latest\n \n - name: Run Re-movery Security Scan\n run: |\n re-movery scan --dir . --exclude \"vendor,node_modules,*.min.js\" --output report.html --format html\n \n - name: Upload Scan Results\n uses: actions/upload-artifact@v2\n with:\n name: security-scan-report\n path: report.html\n`\n\t\n\t// Write content to file\n\treturn os.WriteFile(outputPath, []byte(content), 0644)\n}\n\n// generateGitlabCIFile generates a GitLab CI configuration file\nfunc generateGitlabCIFile(outputPath string) error {\n\t// Create output directory if it doesn't exist\n\tif err := os.MkdirAll(filepath.Dir(outputPath), 0755); err != nil {\n\t\treturn err\n\t}\n\t\n\t// GitLab CI configuration file content\n\tcontent := `stages:\n - security-scan\n\nsecurity-scan:\n stage: security-scan\n image: golang:1.17\n script:\n - go install github.com/re-movery/re-movery@latest\n - re-movery scan --dir . --exclude \"vendor,node_modules,*.min.js\" --output report.html --format html\n artifacts:\n paths:\n - report.html\n expire_in: 1 week\n`\n\t\n\t// Write content to file\n\treturn os.WriteFile(outputPath, []byte(content), 0644)\n}\n\n// generateVSCodeExtensionFiles generates VS Code extension configuration files\nfunc generateVSCodeExtensionFiles(outputPath string) error {\n\t// Create output directory if it doesn't exist\n\tif err := os.MkdirAll(outputPath, 0755); err != nil {\n\t\treturn err\n\t}\n\t\n\t// package.json content\n\tpackageJSON := `{\n \"name\": \"re-movery-vscode\",\n \"displayName\": \"Re-movery Security Scanner\",\n \"description\": \"Security vulnerability scanner for VS Code\",\n \"version\": \"0.1.0\",\n \"engines\": {\n \"vscode\": \"^1.60.0\"\n },\n \"categories\": [\n \"Linters\",\n \"Security\"\n ],\n \"activationEvents\": [\n \"onLanguage:python\",\n \"onLanguage:javascript\",\n \"onCommand:re-movery.scanFile\",\n \"onCommand:re-movery.scanWorkspace\"\n ],\n \"main\": \"./extension.js\",\n \"contributes\": {\n \"commands\": [\n {\n \"command\": \"re-movery.scanFile\",\n \"title\": \"Re-movery: Scan Current File\"\n },\n {\n \"command\": \"re-movery.scanWorkspace\",\n \"title\": \"Re-movery: Scan Workspace\"\n }\n ],\n \"configuration\": {\n \"title\": \"Re-movery\",\n \"properties\": {\n \"re-movery.serverHost\": {\n \"type\": \"string\",\n \"default\": \"localhost\",\n \"description\": \"Host of the Re-movery API server\"\n },\n \"re-movery.serverPort\": {\n \"type\": \"number\",\n \"default\": 8081,\n \"description\": \"Port of the Re-movery API server\"\n },\n \"re-movery.enableBackgroundScanning\": {\n \"type\": \"boolean\",\n \"default\": true,\n \"description\": \"Enable background scanning of files\"\n }\n }\n }\n }\n}\n`\n\t\n\t// extension.js content\n\textensionJS := `const vscode = require('vscode');\nconst path = require('path');\nconst fs = require('fs');\nconst http = require('http');\n\nlet diagnosticCollection;\n\n/**\n * @param {vscode.ExtensionContext} context\n */\nfunction activate(context) {\n console.log('Re-movery extension is now active');\n \n // Create diagnostic collection\n diagnosticCollection = vscode.languages.createDiagnosticCollection('re-movery');\n context.subscriptions.push(diagnosticCollection);\n \n // Register commands\n context.subscriptions.push(\n vscode.commands.registerCommand('re-movery.scanFile', scanCurrentFile),\n vscode.commands.registerCommand('re-movery.scanWorkspace', scanWorkspace)\n );\n \n // Register event handlers\n if (vscode.workspace.getConfiguration('re-movery').get('enableBackgroundScanning')) {\n context.subscriptions.push(\n vscode.workspace.onDidSaveTextDocument(scanDocument),\n vscode.window.onDidChangeActiveTextEditor(editor => {\n if (editor) {\n scanDocument(editor.document);\n }\n })\n );\n }\n}\n\nfunction deactivate() {\n diagnosticCollection.clear();\n}\n\nasync function scanCurrentFile() {\n const editor = vscode.window.activeTextEditor;\n if (!editor) {\n vscode.window.showInformationMessage('No file is currently open');\n return;\n }\n \n await scanDocument(editor.document);\n vscode.window.showInformationMessage('File scan completed');\n}\n\nasync function scanWorkspace() {\n if (!vscode.workspace.workspaceFolders) {\n vscode.window.showInformationMessage('No workspace is open');\n return;\n }\n \n const workspaceFolder = vscode.workspace.workspaceFolders[0].uri.fsPath;\n \n vscode.window.withProgress({\n location: vscode.ProgressLocation.Notification,\n title: 'Scanning workspace for security vulnerabilities',\n cancellable: false\n }, async (progress) => {\n progress.report({ increment: 0 });\n \n try {\n const results = await scanDirectory(workspaceFolder);\n updateDiagnostics(results);\n \n const totalIssues = Object.values(results).reduce((sum, matches) => sum + matches.length, 0);\n vscode.window.showInformationMessage(\\`Workspace scan completed. Found \\${totalIssues} issues.\\`);\n \n progress.report({ increment: 100 });\n } catch (error) {\n vscode.window.showErrorMessage(\\`Error scanning workspace: \\${error.message}\\`);\n }\n });\n}\n\nasync function scanDocument(document) {\n // Check if file type is supported\n if (!isSupportedFileType(document)) {\n return;\n }\n \n try {\n const results = await scanCode(document.getText(), document.fileName);\n updateDiagnosticsForFile(document.uri, results);\n } catch (error) {\n console.error('Error scanning document:', error);\n }\n}\n\nfunction isSupportedFileType(document) {\n const supportedLanguages = ['python', 'javascript'];\n return supportedLanguages.includes(document.languageId);\n}\n\nasync function scanCode(code, filename) {\n const config = vscode.workspace.getConfiguration('re-movery');\n const host = config.get('serverHost');\n const port = config.get('serverPort');\n \n return new Promise((resolve, reject) => {\n const postData = JSON.stringify({\n code: code,\n filename: path.basename(filename)\n });\n \n const options = {\n hostname: host,\n port: port,\n path: '/api/scan/code',\n method: 'POST',\n headers: {\n 'Content-Type': 'application/json',\n 'Content-Length': Buffer.byteLength(postData)\n }\n };\n \n const req = http.request(options, (res) => {\n let data = '';\n \n res.on('data', (chunk) => {\n data += chunk;\n });\n \n res.on('end', () => {\n if (res.statusCode === 200) {\n try {\n const response = JSON.parse(data);\n resolve(response.results || {});\n } catch (error) {\n reject(new Error('Invalid response from server'));\n }\n } else {\n reject(new Error(\\`Server returned status code \\${res.statusCode}\\`));\n }\n });\n });\n \n req.on('error', (error) => {\n reject(new Error(\\`Error connecting to Re-movery server: \\${error.message}\\`));\n });\n \n req.write(postData);\n req.end();\n });\n}\n\nasync function scanDirectory(directory) {\n const config = vscode.workspace.getConfiguration('re-movery');\n const host = config.get('serverHost');\n const port = config.get('serverPort');\n \n return new Promise((resolve, reject) => {\n const postData = JSON.stringify({\n directory: directory\n });\n \n const options = {\n hostname: host,\n port: port,\n path: '/api/scan/directory',\n method: 'POST',\n headers: {\n 'Content-Type': 'application/json',\n 'Content-Length': Buffer.byteLength(postData)\n }\n };\n \n const req = http.request(options, (res) => {\n let data = '';\n \n res.on('data', (chunk) => {\n data += chunk;\n });\n \n res.on('end', () => {\n if (res.statusCode === 200) {\n try {\n const response = JSON.parse(data);\n resolve(response.results || {});\n } catch (error) {\n reject(new Error('Invalid response from server'));\n }\n } else {\n reject(new Error(\\`Server returned status code \\${res.statusCode}\\`));\n }\n });\n });\n \n req.on('error', (error) => {\n reject(new Error(\\`Error connecting to Re-movery server: \\${error.message}\\`));\n });\n \n req.write(postData);\n req.end();\n });\n}\n\nfunction updateDiagnostics(results) {\n // Clear all diagnostics\n diagnosticCollection.clear();\n \n // Update diagnostics for each file\n for (const [filePath, matches] of Object.entries(results)) {\n const uri = vscode.Uri.file(filePath);\n updateDiagnosticsForFile(uri, { [filePath]: matches });\n }\n}\n\nfunction updateDiagnosticsForFile(uri, results) {\n const filePath = uri.fsPath;\n const matches = results[filePath] || [];\n \n if (matches.length === 0) {\n diagnosticCollection.delete(uri);\n return;\n }\n \n const diagnostics = matches.map(match => {\n const range = new vscode.Range(\n match.line - 1, 0,\n match.line - 1, 1000\n );\n \n const severity = getSeverity(match.severity);\n \n return new vscode.Diagnostic(\n range,\n \\`\\${match.name}: \\${match.description}\\`,\n severity\n );\n });\n \n diagnosticCollection.set(uri, diagnostics);\n}\n\nfunction getSeverity(severity) {\n switch (severity.toLowerCase()) {\n case 'high':\n return vscode.DiagnosticSeverity.Error;\n case 'medium':\n return vscode.DiagnosticSeverity.Warning;\n case 'low':\n return vscode.DiagnosticSeverity.Information;\n default:\n return vscode.DiagnosticSeverity.Hint;\n }\n}\n\nmodule.exports = {\n activate,\n deactivate\n};\n`\n\t\n\t// README.md content\n\treadmeMD := `# Re-movery Security Scanner for VS Code\n\nThis extension integrates the Re-movery security scanner into VS Code, providing real-time security vulnerability detection for your code.\n\n## Features\n\n- Scan individual files for security vulnerabilities\n- Scan entire workspaces for security vulnerabilities\n- Real-time scanning as you type\n- Detailed diagnostics with severity levels\n- Integration with the Re-movery API server\n\n## Requirements\n\n- VS Code 1.60.0 or higher\n- Re-movery API server running locally or remotely\n\n## Extension Settings\n\nThis extension contributes the following settings:\n\n* \\`re-movery.serverHost\\`: Host of the Re-movery API server\n* \\`re-movery.serverPort\\`: Port of the Re-movery API server\n* \\`re-movery.enableBackgroundScanning\\`: Enable background scanning of files\n\n## Known Issues\n\n- Currently only supports Python and JavaScript files\n- Requires a running Re-movery API server\n\n## Release Notes\n\n### 0.1.0\n\nInitial release of the Re-movery Security Scanner for VS Code\n`\n\t\n\t// Write files\n\tif err := os.WriteFile(filepath.Join(outputPath, \"package.json\"), []byte(packageJSON), 0644); err != nil {\n\t\treturn err\n\t}\n\t\n\tif err := os.WriteFile(filepath.Join(outputPath, \"extension.js\"), []byte(extensionJS), 0644); err != nil {\n\t\treturn err\n\t}\n\t\n\tif err := os.WriteFile(filepath.Join(outputPath, \"README.md\"), []byte(readmeMD), 0644); err != nil {\n\t\treturn err\n\t}\n\t\n\treturn nil\n} " }, { "path": "go/internal/cmd/root.go", "content": "package cmd\n\nimport (\n\t\"fmt\"\n\t\"os\"\n\n\t\"github.com/spf13/cobra\"\n)\n\nvar rootCmd = &cobra.Command{\n\tUse: \"re-movery\",\n\tShort: \"Re-movery - Security Vulnerability Scanner\",\n\tLong: `Re-movery is a powerful security vulnerability scanner designed to detect \npotential security issues in your codebase. It supports multiple programming \nlanguages and provides various interfaces for scanning and reporting.`,\n\tRun: func(cmd *cobra.Command, args []string) {\n\t\t// If no subcommand is provided, print help\n\t\tcmd.Help()\n\t},\n}\n\n// Execute executes the root command\nfunc Execute() error {\n\treturn rootCmd.Execute()\n}\n\nfunc init() {\n\t// Add global flags\n\trootCmd.PersistentFlags().BoolP(\"verbose\", \"v\", false, \"Enable verbose output\")\n\trootCmd.PersistentFlags().StringP(\"config\", \"c\", \"\", \"Config file path\")\n\n\t// Add subcommands\n\trootCmd.AddCommand(scanCmd)\n\trootCmd.AddCommand(webCmd)\n\trootCmd.AddCommand(serverCmd)\n\trootCmd.AddCommand(generateCmd)\n\trootCmd.AddCommand(versionCmd)\n}\n\n// versionCmd represents the version command\nvar versionCmd = &cobra.Command{\n\tUse: \"version\",\n\tShort: \"Print the version number\",\n\tRun: func(cmd *cobra.Command, args []string) {\n\t\tfmt.Println(\"Re-movery v1.0.0\")\n\t},\n} " }, { "path": "go/internal/cmd/scan.go", "content": "package cmd\n\nimport (\n\t\"fmt\"\n\t\"os\"\n\t\"path/filepath\"\n\t\"strings\"\n\t\"time\"\n\n\t\"github.com/re-movery/re-movery/internal/core\"\n\t\"github.com/re-movery/re-movery/internal/detectors\"\n\t\"github.com/re-movery/re-movery/internal/reporters\"\n\t\"github.com/spf13/cobra\"\n)\n\nvar (\n\tscanFile string\n\tscanDir string\n\texcludePattern string\n\toutputFile string\n\treportFormat string\n\tparallel bool\n\tincremental bool\n\tconfidence float64\n)\n\nvar scanCmd = &cobra.Command{\n\tUse: \"scan\",\n\tShort: \"Scan files or directories for security vulnerabilities\",\n\tLong: `Scan files or directories for security vulnerabilities.\nExamples:\n re-movery scan --file path/to/file.py\n re-movery scan --dir path/to/directory --exclude \"node_modules,*.min.js\"\n re-movery scan --dir path/to/directory --output report.html --format html`,\n\tRun: func(cmd *cobra.Command, args []string) {\n\t\t// Create scanner\n\t\tscanner := core.NewScanner()\n\t\t\n\t\t// Register detectors\n\t\tscanner.RegisterDetector(detectors.NewPythonDetector())\n\t\tscanner.RegisterDetector(detectors.NewJavaScriptDetector())\n\t\t\n\t\t// Set scanner options\n\t\tscanner.SetParallel(parallel)\n\t\tscanner.SetIncremental(incremental)\n\t\tscanner.SetConfidenceThreshold(confidence)\n\t\t\n\t\t// Parse exclude patterns\n\t\tvar excludePatterns []string\n\t\tif excludePattern != \"\" {\n\t\t\texcludePatterns = strings.Split(excludePattern, \",\")\n\t\t\tfor i, pattern := range excludePatterns {\n\t\t\t\texcludePatterns[i] = strings.TrimSpace(pattern)\n\t\t\t}\n\t\t}\n\t\t\n\t\t// Scan file or directory\n\t\tvar results map[string][]core.Match\n\t\tvar err error\n\t\t\n\t\tif scanFile != \"\" {\n\t\t\t// Check if file exists\n\t\t\tif _, err := os.Stat(scanFile); os.IsNotExist(err) {\n\t\t\t\tfmt.Fprintf(os.Stderr, \"Error: File does not exist: %s\\n\", scanFile)\n\t\t\t\tos.Exit(1)\n\t\t\t}\n\t\t\t\n\t\t\t// Scan file\n\t\t\tmatches, err := scanner.ScanFile(scanFile)\n\t\t\tif err != nil {\n\t\t\t\tfmt.Fprintf(os.Stderr, \"Error scanning file: %v\\n\", err)\n\t\t\t\tos.Exit(1)\n\t\t\t}\n\t\t\t\n\t\t\tresults = map[string][]core.Match{\n\t\t\t\tscanFile: matches,\n\t\t\t}\n\t\t} else if scanDir != \"\" {\n\t\t\t// Check if directory exists\n\t\t\tif _, err := os.Stat(scanDir); os.IsNotExist(err) {\n\t\t\t\tfmt.Fprintf(os.Stderr, \"Error: Directory does not exist: %s\\n\", scanDir)\n\t\t\t\tos.Exit(1)\n\t\t\t}\n\t\t\t\n\t\t\t// Scan directory\n\t\t\tresults, err = scanner.ScanDirectory(scanDir, excludePatterns)\n\t\t\tif err != nil {\n\t\t\t\tfmt.Fprintf(os.Stderr, \"Error scanning directory: %v\\n\", err)\n\t\t\t\tos.Exit(1)\n\t\t\t}\n\t\t} else {\n\t\t\tfmt.Fprintf(os.Stderr, \"Error: Please specify a file or directory to scan\\n\")\n\t\t\tcmd.Help()\n\t\t\tos.Exit(1)\n\t\t}\n\t\t\n\t\t// Generate summary\n\t\tsummary := core.GenerateSummary(results)\n\t\t\n\t\t// Print summary to console\n\t\tfmt.Printf(\"Scan completed in %s\\n\", time.Now().Format(time.RFC3339))\n\t\tfmt.Printf(\"Files scanned: %d\\n\", summary.TotalFiles)\n\t\tfmt.Printf(\"Issues found: %d (High: %d, Medium: %d, Low: %d)\\n\",\n\t\t\tsummary.High+summary.Medium+summary.Low, summary.High, summary.Medium, summary.Low)\n\t\t\n\t\t// Generate report if output file is specified\n\t\tif outputFile != \"\" {\n\t\t\t// Create report data\n\t\t\treportData := core.ReportData{\n\t\t\t\tTitle: \"Re-movery Security Scan Report\",\n\t\t\t\tTimestamp: time.Now().Format(time.RFC3339),\n\t\t\t\tResults: results,\n\t\t\t\tSummary: summary,\n\t\t\t}\n\t\t\t\n\t\t\t// Determine report format\n\t\t\tif reportFormat == \"\" {\n\t\t\t\t// Try to determine format from file extension\n\t\t\t\text := strings.ToLower(filepath.Ext(outputFile))\n\t\t\t\tswitch ext {\n\t\t\t\tcase \".html\":\n\t\t\t\t\treportFormat = \"html\"\n\t\t\t\tcase \".json\":\n\t\t\t\t\treportFormat = \"json\"\n\t\t\t\tcase \".xml\":\n\t\t\t\t\treportFormat = \"xml\"\n\t\t\t\tdefault:\n\t\t\t\t\treportFormat = \"html\" // Default to HTML\n\t\t\t\t}\n\t\t\t}\n\t\t\t\n\t\t\t// Generate report\n\t\t\tvar reporter core.Reporter\n\t\t\tswitch strings.ToLower(reportFormat) {\n\t\t\tcase \"html\":\n\t\t\t\treporter = reporters.NewHTMLReporter()\n\t\t\tcase \"json\":\n\t\t\t\treporter = reporters.NewJSONReporter()\n\t\t\tcase \"xml\":\n\t\t\t\treporter = reporters.NewXMLReporter()\n\t\t\tdefault:\n\t\t\t\tfmt.Fprintf(os.Stderr, \"Error: Unsupported report format: %s\\n\", reportFormat)\n\t\t\t\tos.Exit(1)\n\t\t\t}\n\t\t\t\n\t\t\tif err := reporter.GenerateReport(reportData, outputFile); err != nil {\n\t\t\t\tfmt.Fprintf(os.Stderr, \"Error generating report: %v\\n\", err)\n\t\t\t\tos.Exit(1)\n\t\t\t}\n\t\t\t\n\t\t\tfmt.Printf(\"Report generated: %s\\n\", outputFile)\n\t\t}\n\t},\n}\n\nfunc init() {\n\t// Add flags\n\tscanCmd.Flags().StringVar(&scanFile, \"file\", \"\", \"File to scan\")\n\tscanCmd.Flags().StringVar(&scanDir, \"dir\", \"\", \"Directory to scan\")\n\tscanCmd.Flags().StringVar(&excludePattern, \"exclude\", \"\", \"Patterns to exclude (comma separated)\")\n\tscanCmd.Flags().StringVar(&outputFile, \"output\", \"\", \"Output file for the report\")\n\tscanCmd.Flags().StringVar(&reportFormat, \"format\", \"\", \"Report format (html, json, xml)\")\n\tscanCmd.Flags().BoolVar(¶llel, \"parallel\", false, \"Enable parallel processing\")\n\tscanCmd.Flags().BoolVar(&incremental, \"incremental\", false, \"Enable incremental scanning\")\n\tscanCmd.Flags().Float64Var(&confidence, \"confidence\", 0.7, \"Confidence threshold (0.0-1.0)\")\n} " }, { "path": "go/internal/cmd/server.go", "content": "package cmd\n\nimport (\n\t\"fmt\"\n\t\"os\"\n\n\t\"github.com/re-movery/re-movery/internal/api\"\n\t\"github.com/spf13/cobra\"\n)\n\nvar (\n\tserverHost string\n\tserverPort int\n\tserverDebug bool\n)\n\nvar serverCmd = &cobra.Command{\n\tUse: \"server\",\n\tShort: \"Start the API server\",\n\tLong: `Start the API server for Re-movery.\nThe API server provides a RESTful API for scanning files and directories for security vulnerabilities.\n\nExamples:\n re-movery server\n re-movery server --host 0.0.0.0 --port 8081\n re-movery server --debug`,\n\tRun: func(cmd *cobra.Command, args []string) {\n\t\t// Create API server\n\t\tserver := api.NewServer()\n\t\t\n\t\t// Start API server\n\t\taddr := fmt.Sprintf(\"%s:%d\", serverHost, serverPort)\n\t\tfmt.Printf(\"Starting API server at http://%s\\n\", addr)\n\t\t\n\t\tif err := server.Run(serverHost, serverPort, serverDebug); err != nil {\n\t\t\tfmt.Fprintf(os.Stderr, \"Error starting API server: %v\\n\", err)\n\t\t\tos.Exit(1)\n\t\t}\n\t},\n}\n\nfunc init() {\n\t// Add flags\n\tserverCmd.Flags().StringVar(&serverHost, \"host\", \"localhost\", \"Host to bind the API server to\")\n\tserverCmd.Flags().IntVar(&serverPort, \"port\", 8081, \"Port to bind the API server to\")\n\tserverCmd.Flags().BoolVar(&serverDebug, \"debug\", false, \"Enable debug mode\")\n} " }, { "path": "go/internal/cmd/web.go", "content": "package cmd\n\nimport (\n\t\"fmt\"\n\t\"os\"\n\n\t\"github.com/re-movery/re-movery/internal/web\"\n\t\"github.com/spf13/cobra\"\n)\n\nvar (\n\twebHost string\n\twebPort int\n\twebDebug bool\n)\n\nvar webCmd = &cobra.Command{\n\tUse: \"web\",\n\tShort: \"Start the web interface\",\n\tLong: `Start the web interface for Re-movery.\nThe web interface provides a user-friendly way to scan files and directories for security vulnerabilities.\n\nExamples:\n re-movery web\n re-movery web --host 0.0.0.0 --port 8080\n re-movery web --debug`,\n\tRun: func(cmd *cobra.Command, args []string) {\n\t\t// Create web app\n\t\tapp := web.NewApp()\n\t\t\n\t\t// Start web server\n\t\taddr := fmt.Sprintf(\"%s:%d\", webHost, webPort)\n\t\tfmt.Printf(\"Starting web server at http://%s\\n\", addr)\n\t\t\n\t\tif err := app.Run(webHost, webPort, webDebug); err != nil {\n\t\t\tfmt.Fprintf(os.Stderr, \"Error starting web server: %v\\n\", err)\n\t\t\tos.Exit(1)\n\t\t}\n\t},\n}\n\nfunc init() {\n\t// Add flags\n\twebCmd.Flags().StringVar(&webHost, \"host\", \"localhost\", \"Host to bind the web server to\")\n\twebCmd.Flags().IntVar(&webPort, \"port\", 8080, \"Port to bind the web server to\")\n\twebCmd.Flags().BoolVar(&webDebug, \"debug\", false, \"Enable debug mode\")\n} " }, { "path": "go/internal/config/config.go", "content": "package config\n\nimport (\n \"github.com/spf13/viper\"\n)\n\n// Config represents the application configuration\ntype Config struct {\n Processing ProcessingConfig `mapstructure:\"processing\"`\n Detector DetectorConfig `mapstructure:\"detector\"`\n Logging LoggingConfig `mapstructure:\"logging\"`\n Security SecurityConfig `mapstructure:\"security\"`\n}\n\n// ProcessingConfig contains processing-related configuration\ntype ProcessingConfig struct {\n NumWorkers int `mapstructure:\"num_workers\"`\n MaxMemoryGB float64 `mapstructure:\"max_memory_gb\"`\n ChunkSizeMB int `mapstructure:\"chunk_size_mb\"`\n EnableCache bool `mapstructure:\"enable_cache\"`\n CacheSize int `mapstructure:\"cache_size\"`\n Languages []string `mapstructure:\"languages\"`\n}\n\n// DetectorConfig contains detector-related configuration\ntype DetectorConfig struct {\n MinSimilarity float64 `mapstructure:\"min_similarity\"`\n EditDistance int `mapstructure:\"edit_distance\"`\n ContextLines int `mapstructure:\"context_lines\"`\n ASTDepth int `mapstructure:\"ast_depth\"`\n CFGNodes int `mapstructure:\"cfg_nodes\"`\n ReportFormat []string `mapstructure:\"report_format\"`\n ExcludePatterns []string `mapstructure:\"exclude_patterns\"`\n}\n\n// LoggingConfig contains logging-related configuration\ntype LoggingConfig struct {\n Level string `mapstructure:\"level\"`\n File string `mapstructure:\"file\"`\n Format string `mapstructure:\"format\"`\n EnableProfiling bool `mapstructure:\"enable_profiling\"`\n ShowProgress bool `mapstructure:\"show_progress\"`\n}\n\n// SecurityConfig contains security-related configuration\ntype SecurityConfig struct {\n MaxFileSizeMB int `mapstructure:\"max_file_size_mb\"`\n AllowedSchemes []string `mapstructure:\"allowed_schemes\"`\n EnableSandbox bool `mapstructure:\"enable_sandbox\"`\n RequireAuth bool `mapstructure:\"require_auth\"`\n RateLimitPerHour int `mapstructure:\"rate_limit_per_hour\"`\n}\n\n// LoadConfig loads the configuration from file\nfunc LoadConfig(configFile string) (*Config, error) {\n viper.SetConfigFile(configFile)\n viper.SetConfigType(\"json\")\n\n if err := viper.ReadInConfig(); err != nil {\n return nil, err\n }\n\n var config Config\n if err := viper.Unmarshal(&config); err != nil {\n return nil, err\n }\n\n return &config, nil\n}\n\n// SetDefaults sets default configuration values\nfunc SetDefaults() {\n viper.SetDefault(\"processing.num_workers\", 4)\n viper.SetDefault(\"processing.max_memory_gb\", 8)\n viper.SetDefault(\"processing.chunk_size_mb\", 1)\n viper.SetDefault(\"processing.enable_cache\", true)\n viper.SetDefault(\"processing.cache_size\", 1000)\n viper.SetDefault(\"processing.languages\", []string{\"go\", \"java\", \"python\", \"javascript\"})\n\n viper.SetDefault(\"detector.min_similarity\", 0.8)\n viper.SetDefault(\"detector.edit_distance\", 3)\n viper.SetDefault(\"detector.context_lines\", 3)\n viper.SetDefault(\"detector.ast_depth\", 5)\n viper.SetDefault(\"detector.cfg_nodes\", 100)\n viper.SetDefault(\"detector.report_format\", []string{\"html\", \"json\"})\n\n viper.SetDefault(\"logging.level\", \"info\")\n viper.SetDefault(\"logging.format\", \"text\")\n viper.SetDefault(\"logging.enable_profiling\", false)\n viper.SetDefault(\"logging.show_progress\", true)\n\n viper.SetDefault(\"security.max_file_size_mb\", 10)\n viper.SetDefault(\"security.enable_sandbox\", true)\n viper.SetDefault(\"security.require_auth\", false)\n viper.SetDefault(\"security.rate_limit_per_hour\", 1000)\n} " }, { "path": "go/internal/core/config.go", "content": "package core\n\nimport (\n\t\"encoding/json\"\n\t\"fmt\"\n\t\"io/ioutil\"\n\t\"os\"\n\t\"path/filepath\"\n\t\"strings\"\n\n\t\"gopkg.in/yaml.v3\"\n)\n\n// Config 表示应用程序配置\ntype Config struct {\n\tScanner ScannerConfig `json:\"scanner\" yaml:\"scanner\"`\n\tWeb WebConfig `json:\"web\" yaml:\"web\"`\n\tServer ServerConfig `json:\"server\" yaml:\"server\"`\n}\n\n// ScannerConfig 表示扫描器配置\ntype ScannerConfig struct {\n\tParallel bool `json:\"parallel\" yaml:\"parallel\"`\n\tIncremental bool `json:\"incremental\" yaml:\"incremental\"`\n\tConfidenceThreshold float64 `json:\"confidenceThreshold\" yaml:\"confidenceThreshold\"`\n\tExcludePatterns []string `json:\"excludePatterns\" yaml:\"excludePatterns\"`\n}\n\n// WebConfig 表示Web界面配置\ntype WebConfig struct {\n\tHost string `json:\"host\" yaml:\"host\"`\n\tPort int `json:\"port\" yaml:\"port\"`\n\tDebug bool `json:\"debug\" yaml:\"debug\"`\n}\n\n// ServerConfig 表示API服务器配置\ntype ServerConfig struct {\n\tHost string `json:\"host\" yaml:\"host\"`\n\tPort int `json:\"port\" yaml:\"port\"`\n\tDebug bool `json:\"debug\" yaml:\"debug\"`\n}\n\n// NewConfig 创建一个新的配置对象，使用默认值\nfunc NewConfig() *Config {\n\treturn &Config{\n\t\tScanner: ScannerConfig{\n\t\t\tParallel: false,\n\t\t\tIncremental: false,\n\t\t\tConfidenceThreshold: 0.7,\n\t\t\tExcludePatterns: []string{},\n\t\t},\n\t\tWeb: WebConfig{\n\t\t\tHost: \"localhost\",\n\t\t\tPort: 8080,\n\t\t\tDebug: false,\n\t\t},\n\t\tServer: ServerConfig{\n\t\t\tHost: \"localhost\",\n\t\t\tPort: 8081,\n\t\t\tDebug: false,\n\t\t},\n\t}\n}\n\n// LoadConfig 从文件加载配置\nfunc LoadConfig(configPath string) (*Config, error) {\n\t// 如果未指定配置文件，则使用默认配置\n\tif configPath == \"\" {\n\t\treturn NewConfig(), nil\n\t}\n\n\t// 检查文件是否存在\n\tif _, err := os.Stat(configPath); os.IsNotExist(err) {\n\t\treturn nil, fmt.Errorf(\"配置文件不存在: %s\", configPath)\n\t}\n\n\t// 读取文件内容\n\tdata, err := ioutil.ReadFile(configPath)\n\tif err != nil {\n\t\treturn nil, err\n\t}\n\n\t// 根据文件扩展名解析配置\n\tconfig := NewConfig()\n\text := strings.ToLower(filepath.Ext(configPath))\n\tswitch ext {\n\tcase \".json\":\n\t\tif err := json.Unmarshal(data, config); err != nil {\n\t\t\treturn nil, err\n\t\t}\n\tcase \".yaml\", \".yml\":\n\t\tif err := yaml.Unmarshal(data, config); err != nil {\n\t\t\treturn nil, err\n\t\t}\n\tdefault:\n\t\treturn nil, fmt.Errorf(\"不支持的配置文件格式: %s\", ext)\n\t}\n\n\treturn config, nil\n}\n\n// SaveConfig 将配置保存到文件\nfunc SaveConfig(config *Config, configPath string) error {\n\t// 创建输出目录（如果不存在）\n\toutputDir := filepath.Dir(configPath)\n\tif err := os.MkdirAll(outputDir, 0755); err != nil {\n\t\treturn err\n\t}\n\n\t// 根据文件扩展名序列化配置\n\tvar data []byte\n\tvar err error\n\text := strings.ToLower(filepath.Ext(configPath))\n\tswitch ext {\n\tcase \".json\":\n\t\tdata, err = json.MarshalIndent(config, \"\", \" \")\n\t\tif err != nil {\n\t\t\treturn err\n\t\t}\n\tcase \".yaml\", \".yml\":\n\t\tdata, err = yaml.Marshal(config)\n\t\tif err != nil {\n\t\t\treturn err\n\t\t}\n\tdefault:\n\t\treturn fmt.Errorf(\"不支持的配置文件格式: %s\", ext)\n\t}\n\n\t// 写入文件\n\treturn ioutil.WriteFile(configPath, data, 0644)\n}\n\n// ApplyToScanner 将配置应用到扫描器\nfunc (c *Config) ApplyToScanner(scanner *Scanner) {\n\tscanner.SetParallel(c.Scanner.Parallel)\n\tscanner.SetIncremental(c.Scanner.Incremental)\n\tscanner.SetConfidenceThreshold(c.Scanner.ConfidenceThreshold)\n} " }, { "path": "go/internal/core/config_test.go", "content": "package core\n\nimport (\n\t\"io/ioutil\"\n\t\"os\"\n\t\"path/filepath\"\n\t\"testing\"\n\n\t\"github.com/stretchr/testify/assert\"\n)\n\n// 测试创建新配置\nfunc TestNewConfig(t *testing.T) {\n\tconfig := NewConfig()\n\tassert.NotNil(t, config)\n\t\n\t// 检查默认值\n\tassert.False(t, config.Scanner.Parallel)\n\tassert.False(t, config.Scanner.Incremental)\n\tassert.Equal(t, 0.7, config.Scanner.ConfidenceThreshold)\n\tassert.Equal(t, \"localhost\", config.Web.Host)\n\tassert.Equal(t, 8080, config.Web.Port)\n\tassert.False(t, config.Web.Debug)\n\tassert.Equal(t, \"localhost\", config.Server.Host)\n\tassert.Equal(t, 8081, config.Server.Port)\n\tassert.False(t, config.Server.Debug)\n}\n\n// 测试加载JSON配置\nfunc TestLoadConfigJSON(t *testing.T) {\n\t// 创建临时配置文件\n\tcontent := []byte(`{\n\t\t\"scanner\": {\n\t\t\t\"parallel\": true,\n\t\t\t\"incremental\": true,\n\t\t\t\"confidenceThreshold\": 0.8,\n\t\t\t\"excludePatterns\": [\"node_modules\", \"*.min.js\"]\n\t\t},\n\t\t\"web\": {\n\t\t\t\"host\": \"0.0.0.0\",\n\t\t\t\"port\": 9090,\n\t\t\t\"debug\": true\n\t\t},\n\t\t\"server\": {\n\t\t\t\"host\": \"0.0.0.0\",\n\t\t\t\"port\": 9091,\n\t\t\t\"debug\": true\n\t\t}\n\t}`)\n\t\n\ttmpfile, err := ioutil.TempFile(\"\", \"config-*.json\")\n\tassert.NoError(t, err)\n\tdefer os.Remove(tmpfile.Name())\n\t\n\t_, err = tmpfile.Write(content)\n\tassert.NoError(t, err)\n\terr = tmpfile.Close()\n\tassert.NoError(t, err)\n\t\n\t// 加载配置\n\tconfig, err := LoadConfig(tmpfile.Name())\n\tassert.NoError(t, err)\n\tassert.NotNil(t, config)\n\t\n\t// 检查加载的值\n\tassert.True(t, config.Scanner.Parallel)\n\tassert.True(t, config.Scanner.Incremental)\n\tassert.Equal(t, 0.8, config.Scanner.ConfidenceThreshold)\n\tassert.Equal(t, []string{\"node_modules\", \"*.min.js\"}, config.Scanner.ExcludePatterns)\n\tassert.Equal(t, \"0.0.0.0\", config.Web.Host)\n\tassert.Equal(t, 9090, config.Web.Port)\n\tassert.True(t, config.Web.Debug)\n\tassert.Equal(t, \"0.0.0.0\", config.Server.Host)\n\tassert.Equal(t, 9091, config.Server.Port)\n\tassert.True(t, config.Server.Debug)\n}\n\n// 测试加载YAML配置\nfunc TestLoadConfigYAML(t *testing.T) {\n\t// 创建临时配置文件\n\tcontent := []byte(`scanner:\n parallel: true\n incremental: true\n confidenceThreshold: 0.8\n excludePatterns:\n - node_modules\n - \"*.min.js\"\nweb:\n host: 0.0.0.0\n port: 9090\n debug: true\nserver:\n host: 0.0.0.0\n port: 9091\n debug: true\n`)\n\t\n\ttmpfile, err := ioutil.TempFile(\"\", \"config-*.yaml\")\n\tassert.NoError(t, err)\n\tdefer os.Remove(tmpfile.Name())\n\t\n\t_, err = tmpfile.Write(content)\n\tassert.NoError(t, err)\n\terr = tmpfile.Close()\n\tassert.NoError(t, err)\n\t\n\t// 加载配置\n\tconfig, err := LoadConfig(tmpfile.Name())\n\tassert.NoError(t, err)\n\tassert.NotNil(t, config)\n\t\n\t// 检查加载的值\n\tassert.True(t, config.Scanner.Parallel)\n\tassert.True(t, config.Scanner.Incremental)\n\tassert.Equal(t, 0.8, config.Scanner.ConfidenceThreshold)\n\tassert.Equal(t, []string{\"node_modules\", \"*.min.js\"}, config.Scanner.ExcludePatterns)\n\tassert.Equal(t, \"0.0.0.0\", config.Web.Host)\n\tassert.Equal(t, 9090, config.Web.Port)\n\tassert.True(t, config.Web.Debug)\n\tassert.Equal(t, \"0.0.0.0\", config.Server.Host)\n\tassert.Equal(t, 9091, config.Server.Port)\n\tassert.True(t, config.Server.Debug)\n}\n\n// 测试保存配置\nfunc TestSaveConfig(t *testing.T) {\n\t// 创建配置\n\tconfig := NewConfig()\n\tconfig.Scanner.Parallel = true\n\tconfig.Scanner.Incremental = true\n\tconfig.Scanner.ConfidenceThreshold = 0.8\n\tconfig.Scanner.ExcludePatterns = []string{\"node_modules\", \"*.min.js\"}\n\tconfig.Web.Host = \"0.0.0.0\"\n\tconfig.Web.Port = 9090\n\tconfig.Web.Debug = true\n\tconfig.Server.Host = \"0.0.0.0\"\n\tconfig.Server.Port = 9091\n\tconfig.Server.Debug = true\n\t\n\t// 创建临时文件路径\n\ttmpdir, err := ioutil.TempDir(\"\", \"config-test\")\n\tassert.NoError(t, err)\n\tdefer os.RemoveAll(tmpdir)\n\t\n\t// 保存JSON配置\n\tjsonPath := filepath.Join(tmpdir, \"config.json\")\n\terr = SaveConfig(config, jsonPath)\n\tassert.NoError(t, err)\n\t\n\t// 保存YAML配置\n\tyamlPath := filepath.Join(tmpdir, \"config.yaml\")\n\terr = SaveConfig(config, yamlPath)\n\tassert.NoError(t, err)\n\t\n\t// 重新加载JSON配置\n\tjsonConfig, err := LoadConfig(jsonPath)\n\tassert.NoError(t, err)\n\tassert.Equal(t, config, jsonConfig)\n\t\n\t// 重新加载YAML配置\n\tyamlConfig, err := LoadConfig(yamlPath)\n\tassert.NoError(t, err)\n\tassert.Equal(t, config, yamlConfig)\n}\n\n// 测试应用配置到扫描器\nfunc TestApplyToScanner(t *testing.T) {\n\t// 创建配置\n\tconfig := NewConfig()\n\tconfig.Scanner.Parallel = true\n\tconfig.Scanner.Incremental = true\n\tconfig.Scanner.ConfidenceThreshold = 0.8\n\t\n\t// 创建扫描器\n\tscanner := NewScanner()\n\t\n\t// 应用配置\n\tconfig.ApplyToScanner(scanner)\n\t\n\t// 检查扫描器设置\n\tassert.True(t, scanner.IsParallel())\n\tassert.True(t, scanner.IsIncremental())\n\tassert.Equal(t, 0.8, scanner.confidenceThreshold)\n} " }, { "path": "go/internal/core/models.go", "content": "package core\n\nimport (\n\t\"time\"\n)\n\n// Signature represents a vulnerability signature\ntype Signature struct {\n\tID string `json:\"id\"`\n\tName string `json:\"name\"`\n\tSeverity string `json:\"severity\"`\n\tDescription string `json:\"description\"`\n\tCodePatterns []string `json:\"codePatterns\"`\n\tReferences []string `json:\"references\"`\n}\n\n// Match represents a vulnerability match\ntype Match struct {\n\tSignature Signature `json:\"signature\"`\n\tFilePath string `json:\"filePath\"`\n\tLineNumber int `json:\"lineNumber\"`\n\tMatchedCode string `json:\"matchedCode\"`\n\tConfidence float64 `json:\"confidence\"`\n}\n\n// Summary represents a summary of scan results\ntype Summary struct {\n\tTotalFiles int `json:\"totalFiles\"`\n\tHigh int `json:\"high\"`\n\tMedium int `json:\"medium\"`\n\tLow int `json:\"low\"`\n\tVulnerabilities map[string]int `json:\"vulnerabilities\"`\n}\n\n// ReportData represents data for a report\ntype ReportData struct {\n\tTitle string `json:\"title\"`\n\tTimestamp string `json:\"timestamp\"`\n\tResults map[string][]Match `json:\"results\"`\n\tSummary Summary `json:\"summary\"`\n}\n\n// Reporter is an interface for report generators\ntype Reporter interface {\n\tGenerateReport(data ReportData, outputPath string) error\n}\n\n// Detector is an interface for vulnerability detectors\ntype Detector interface {\n\tName() string\n\tSupportedLanguages() []string\n\tDetectFile(filePath string) ([]Match, error)\n\tDetectCode(code string, filePath string) ([]Match, error)\n}\n\n// GenerateSummary generates a summary from scan results\nfunc GenerateSummary(results map[string][]Match) Summary {\n\tsummary := Summary{\n\t\tTotalFiles: len(results),\n\t\tVulnerabilities: make(map[string]int),\n\t}\n\n\tfor _, matches := range results {\n\t\tfor _, match := range matches {\n\t\t\tswitch match.Signature.Severity {\n\t\t\tcase \"high\":\n\t\t\t\tsummary.High++\n\t\t\tcase \"medium\":\n\t\t\t\tsummary.Medium++\n\t\t\tcase \"low\":\n\t\t\t\tsummary.Low++\n\t\t\t}\n\n\t\t\t// Count vulnerabilities by name\n\t\t\tsummary.Vulnerabilities[match.Signature.Name]++\n\t\t}\n\t}\n\n\treturn summary\n} " }, { "path": "go/internal/core/scanner.go", "content": "package core\n\nimport (\n\t\"fmt\"\n\t\"os\"\n\t\"path/filepath\"\n\t\"strings\"\n\t\"sync\"\n)\n\n// Scanner is a vulnerability scanner\ntype Scanner struct {\n\tdetectors []Detector\n\tparallel bool\n\tincremental bool\n\tconfidenceThreshold float64\n\tcache map[string][]Match\n\tcacheMutex sync.RWMutex\n}\n\n// NewScanner creates a new scanner\nfunc NewScanner() *Scanner {\n\treturn &Scanner{\n\t\tdetectors: []Detector{},\n\t\tparallel: false,\n\t\tincremental: false,\n\t\tconfidenceThreshold: 0.7,\n\t\tcache: make(map[string][]Match),\n\t}\n}\n\n// RegisterDetector registers a detector\nfunc (s *Scanner) RegisterDetector(detector Detector) {\n\ts.detectors = append(s.detectors, detector)\n}\n\n// SetParallel sets whether to use parallel processing\nfunc (s *Scanner) SetParallel(parallel bool) {\n\ts.parallel = parallel\n}\n\n// IsParallel returns whether parallel processing is enabled\nfunc (s *Scanner) IsParallel() bool {\n\treturn s.parallel\n}\n\n// SetIncremental sets whether to use incremental scanning\nfunc (s *Scanner) SetIncremental(incremental bool) {\n\ts.incremental = incremental\n}\n\n// IsIncremental returns whether incremental scanning is enabled\nfunc (s *Scanner) IsIncremental() bool {\n\treturn s.incremental\n}\n\n// SetConfidenceThreshold sets the confidence threshold\nfunc (s *Scanner) SetConfidenceThreshold(threshold float64) {\n\ts.confidenceThreshold = threshold\n}\n\n// SupportedLanguages returns the list of supported languages\nfunc (s *Scanner) SupportedLanguages() []string {\n\tlanguages := []string{}\n\tfor _, detector := range s.detectors {\n\t\tlanguages = append(languages, detector.SupportedLanguages()...)\n\t}\n\treturn languages\n}\n\n// ScanFile scans a file for vulnerabilities\nfunc (s *Scanner) ScanFile(filePath string) ([]Match, error) {\n\t// Check if file exists\n\tif _, err := os.Stat(filePath); os.IsNotExist(err) {\n\t\treturn nil, fmt.Errorf(\"file does not exist: %s\", filePath)\n\t}\n\n\t// Check if file is in cache\n\tif s.incremental {\n\t\ts.cacheMutex.RLock()\n\t\tif matches, ok := s.cache[filePath]; ok {\n\t\t\ts.cacheMutex.RUnlock()\n\t\t\treturn matches, nil\n\t\t}\n\t\ts.cacheMutex.RUnlock()\n\t}\n\n\t// Scan file with each detector\n\tvar allMatches []Match\n\tfor _, detector := range s.detectors {\n\t\tmatches, err := detector.DetectFile(filePath)\n\t\tif err != nil {\n\t\t\treturn nil, err\n\t\t}\n\n\t\t// Filter matches by confidence threshold\n\t\tfor _, match := range matches {\n\t\t\tif match.Confidence >= s.confidenceThreshold {\n\t\t\t\tallMatches = append(allMatches, match)\n\t\t\t}\n\t\t}\n\t}\n\n\t// Update cache\n\tif s.incremental {\n\t\ts.cacheMutex.Lock()\n\t\ts.cache[filePath] = allMatches\n\t\ts.cacheMutex.Unlock()\n\t}\n\n\treturn allMatches, nil\n}\n\n// ScanDirectory scans a directory for vulnerabilities\nfunc (s *Scanner) ScanDirectory(dirPath string, excludePatterns []string) (map[string][]Match, error) {\n\t// Check if directory exists\n\tif _, err := os.Stat(dirPath); os.IsNotExist(err) {\n\t\treturn nil, fmt.Errorf(\"directory does not exist: %s\", dirPath)\n\t}\n\n\t// Collect files to scan\n\tvar filesToScan []string\n\terr := filepath.Walk(dirPath, func(path string, info os.FileInfo, err error) error {\n\t\tif err != nil {\n\t\t\treturn err\n\t\t}\n\n\t\t// Skip directories\n\t\tif info.IsDir() {\n\t\t\t// Check if directory should be excluded\n\t\t\tfor _, pattern := range excludePatterns {\n\t\t\t\tif matched, _ := filepath.Match(pattern, info.Name()); matched {\n\t\t\t\t\treturn filepath.SkipDir\n\t\t\t\t}\n\t\t\t}\n\t\t\treturn nil\n\t\t}\n\n\t\t// Check if file should be excluded\n\t\tfor _, pattern := range excludePatterns {\n\t\t\tif matched, _ := filepath.Match(pattern, info.Name()); matched {\n\t\t\t\treturn nil\n\t\t\t}\n\t\t}\n\n\t\t// Check if file extension is supported\n\t\text := strings.ToLower(filepath.Ext(path))\n\t\tif ext == \"\" {\n\t\t\treturn nil\n\t\t}\n\n\t\t// Remove the dot from the extension\n\t\text = ext[1:]\n\n\t\t// Check if any detector supports this file type\n\t\tfor _, detector := range s.detectors {\n\t\t\tfor _, lang := range detector.SupportedLanguages() {\n\t\t\t\tif lang == ext {\n\t\t\t\t\tfilesToScan = append(filesToScan, path)\n\t\t\t\t\treturn nil\n\t\t\t\t}\n\t\t\t}\n\t\t}\n\n\t\treturn nil\n\t})\n\n\tif err != nil {\n\t\treturn nil, err\n\t}\n\n\t// Scan files\n\tresults := make(map[string][]Match)\n\tif s.parallel {\n\t\t// Parallel scanning\n\t\tvar wg sync.WaitGroup\n\t\tresultsMutex := sync.Mutex{}\n\n\t\tfor _, file := range filesToScan {\n\t\t\twg.Add(1)\n\t\t\tgo func(file string) {\n\t\t\t\tdefer wg.Done()\n\n\t\t\t\tmatches, err := s.ScanFile(file)\n\t\t\t\tif err != nil {\n\t\t\t\t\t// Log error but continue\n\t\t\t\t\tfmt.Fprintf(os.Stderr, \"Error scanning file %s: %v\\n\", file, err)\n\t\t\t\t\treturn\n\t\t\t\t}\n\n\t\t\t\tif len(matches) > 0 {\n\t\t\t\t\tresultsMutex.Lock()\n\t\t\t\t\tresults[file] = matches\n\t\t\t\t\tresultsMutex.Unlock()\n\t\t\t\t}\n\t\t\t}(file)\n\t\t}\n\n\t\twg.Wait()\n\t} else {\n\t\t// Sequential scanning\n\t\tfor _, file := range filesToScan {\n\t\t\tmatches, err := s.ScanFile(file)\n\t\t\tif err != nil {\n\t\t\t\t// Log error but continue\n\t\t\t\tfmt.Fprintf(os.Stderr, \"Error scanning file %s: %v\\n\", file, err)\n\t\t\t\tcontinue\n\t\t\t}\n\n\t\t\tif len(matches) > 0 {\n\t\t\t\tresults[file] = matches\n\t\t\t}\n\t\t}\n\t}\n\n\treturn results, nil\n} " }, { "path": "go/internal/core/scanner_test.go", "content": "package core\n\nimport (\n\t\"io/ioutil\"\n\t\"os\"\n\t\"path/filepath\"\n\t\"testing\"\n\n\t\"github.com/stretchr/testify/assert\"\n)\n\n// 测试扫描器创建\nfunc TestNewScanner(t *testing.T) {\n\tscanner := NewScanner()\n\tassert.NotNil(t, scanner)\n\tassert.False(t, scanner.IsParallel())\n\tassert.False(t, scanner.IsIncremental())\n}\n\n// 测试设置并行处理\nfunc TestSetParallel(t *testing.T) {\n\tscanner := NewScanner()\n\tassert.False(t, scanner.IsParallel())\n\t\n\tscanner.SetParallel(true)\n\tassert.True(t, scanner.IsParallel())\n\t\n\tscanner.SetParallel(false)\n\tassert.False(t, scanner.IsParallel())\n}\n\n// 测试设置增量扫描\nfunc TestSetIncremental(t *testing.T) {\n\tscanner := NewScanner()\n\tassert.False(t, scanner.IsIncremental())\n\t\n\tscanner.SetIncremental(true)\n\tassert.True(t, scanner.IsIncremental())\n\t\n\tscanner.SetIncremental(false)\n\tassert.False(t, scanner.IsIncremental())\n}\n\n// 测试注册检测器\nfunc TestRegisterDetector(t *testing.T) {\n\tscanner := NewScanner()\n\t\n\t// 创建模拟检测器\n\tdetector := &mockDetector{}\n\t\n\t// 注册检测器\n\tscanner.RegisterDetector(detector)\n\t\n\t// 检查支持的语言\n\tlanguages := scanner.SupportedLanguages()\n\tassert.Contains(t, languages, \"mock\")\n}\n\n// 测试扫描文件\nfunc TestScanFile(t *testing.T) {\n\t// 创建临时文件\n\tcontent := []byte(\"print(eval('1+1'))\")\n\ttmpfile, err := ioutil.TempFile(\"\", \"example.py\")\n\tassert.NoError(t, err)\n\tdefer os.Remove(tmpfile.Name())\n\t\n\t_, err = tmpfile.Write(content)\n\tassert.NoError(t, err)\n\terr = tmpfile.Close()\n\tassert.NoError(t, err)\n\t\n\t// 创建扫描器和模拟检测器\n\tscanner := NewScanner()\n\tdetector := &mockDetector{}\n\tscanner.RegisterDetector(detector)\n\t\n\t// 扫描文件\n\tmatches, err := scanner.ScanFile(tmpfile.Name())\n\tassert.NoError(t, err)\n\tassert.Len(t, matches, 1)\n\tassert.Equal(t, \"MOCK001\", matches[0].Signature.ID)\n}\n\n// 测试扫描目录\nfunc TestScanDirectory(t *testing.T) {\n\t// 创建临时目录\n\ttmpdir, err := ioutil.TempDir(\"\", \"example\")\n\tassert.NoError(t, err)\n\tdefer os.RemoveAll(tmpdir)\n\t\n\t// 创建测试文件\n\tfile1 := filepath.Join(tmpdir, \"test1.py\")\n\terr = ioutil.WriteFile(file1, []byte(\"print(eval('1+1'))\"), 0644)\n\tassert.NoError(t, err)\n\t\n\tfile2 := filepath.Join(tmpdir, \"test2.py\")\n\terr = ioutil.WriteFile(file2, []byte(\"print('Hello')\"), 0644)\n\tassert.NoError(t, err)\n\t\n\t// 创建扫描器和模拟检测器\n\tscanner := NewScanner()\n\tdetector := &mockDetector{}\n\tscanner.RegisterDetector(detector)\n\t\n\t// 扫描目录\n\tresults, err := scanner.ScanDirectory(tmpdir, nil)\n\tassert.NoError(t, err)\n\tassert.Len(t, results, 2)\n\t\n\t// 检查结果\n\tassert.Contains(t, results, file1)\n\tassert.Contains(t, results, file2)\n\tassert.Len(t, results[file1], 1)\n\tassert.Len(t, results[file2], 1)\n}\n\n// 测试生成摘要\nfunc TestGenerateSummary(t *testing.T) {\n\t// 创建测试数据\n\tresults := map[string][]Match{\n\t\t\"file1.py\": {\n\t\t\t{\n\t\t\t\tSignature: Signature{\n\t\t\t\t\tID: \"PY001\",\n\t\t\t\t\tName: \"Dangerous eval() usage\",\n\t\t\t\t\tSeverity: \"high\",\n\t\t\t\t},\n\t\t\t},\n\t\t},\n\t\t\"file2.py\": {\n\t\t\t{\n\t\t\t\tSignature: Signature{\n\t\t\t\t\tID: \"PY002\",\n\t\t\t\t\tName: \"Dangerous exec() usage\",\n\t\t\t\t\tSeverity: \"high\",\n\t\t\t\t},\n\t\t\t},\n\t\t\t{\n\t\t\t\tSignature: Signature{\n\t\t\t\t\tID: \"PY005\",\n\t\t\t\t\tName: \"Insecure random number generation\",\n\t\t\t\t\tSeverity: \"medium\",\n\t\t\t\t},\n\t\t\t},\n\t\t},\n\t\t\"file3.py\": {\n\t\t\t{\n\t\t\t\tSignature: Signature{\n\t\t\t\t\tID: \"PY008\",\n\t\t\t\t\tName: \"Temporary file creation risk\",\n\t\t\t\t\tSeverity: \"medium\",\n\t\t\t\t},\n\t\t\t},\n\t\t\t{\n\t\t\t\tSignature: Signature{\n\t\t\t\t\tID: \"PY010\",\n\t\t\t\t\tName: \"Debug mode enabled\",\n\t\t\t\t\tSeverity: \"medium\",\n\t\t\t\t},\n\t\t\t},\n\t\t\t{\n\t\t\t\tSignature: Signature{\n\t\t\t\t\tID: \"PY012\",\n\t\t\t\t\tName: \"Bare except block\",\n\t\t\t\t\tSeverity: \"low\",\n\t\t\t\t},\n\t\t\t},\n\t\t},\n\t}\n\t\n\t// 生成摘要\n\tsummary := GenerateSummary(results)\n\t\n\t// 检查摘要\n\tassert.Equal(t, 3, summary.TotalFiles)\n\tassert.Equal(t, 2, summary.High)\n\tassert.Equal(t, 3, summary.Medium)\n\tassert.Equal(t, 1, summary.Low)\n\t\n\t// 检查漏洞计数\n\tassert.Equal(t, 1, summary.Vulnerabilities[\"Dangerous eval() usage\"])\n\tassert.Equal(t, 1, summary.Vulnerabilities[\"Dangerous exec() usage\"])\n\tassert.Equal(t, 1, summary.Vulnerabilities[\"Insecure random number generation\"])\n\tassert.Equal(t, 1, summary.Vulnerabilities[\"Temporary file creation risk\"])\n\tassert.Equal(t, 1, summary.Vulnerabilities[\"Debug mode enabled\"])\n\tassert.Equal(t, 1, summary.Vulnerabilities[\"Bare except block\"])\n}\n\n// 模拟检测器\ntype mockDetector struct{}\n\nfunc (d *mockDetector) Name() string {\n\treturn \"mock\"\n}\n\nfunc (d *mockDetector) SupportedLanguages() []string {\n\treturn []string{\"mock\", \"py\", \"python\"}\n}\n\nfunc (d *mockDetector) DetectFile(filePath string) ([]Match, error) {\n\treturn []Match{\n\t\t{\n\t\t\tSignature: Signature{\n\t\t\t\tID: \"MOCK001\",\n\t\t\t\tName: \"Mock vulnerability\",\n\t\t\t\tSeverity: \"high\",\n\t\t\t\tDescription: \"This is a mock vulnerability\",\n\t\t\t},\n\t\t\tFilePath: filePath,\n\t\t\tLineNumber: 1,\n\t\t\tMatchedCode: \"mock code\",\n\t\t\tConfidence: 0.9,\n\t\t},\n\t}, nil\n}\n\nfunc (d *mockDetector) DetectCode(code string, filePath string) ([]Match, error) {\n\treturn []Match{\n\t\t{\n\t\t\tSignature: Signature{\n\t\t\t\tID: \"MOCK001\",\n\t\t\t\tName: \"Mock vulnerability\",\n\t\t\t\tSeverity: \"high\",\n\t\t\t\tDescription: \"This is a mock vulnerability\",\n\t\t\t},\n\t\t\tFilePath: filePath,\n\t\t\tLineNumber: 1,\n\t\t\tMatchedCode: code,\n\t\t\tConfidence: 0.9,\n\t\t},\n\t}, nil\n} " }, { "path": "go/internal/detectors/javascript.go", "content": "package detectors\n\nimport (\n\t\"bufio\"\n\t\"io/ioutil\"\n\t\"os\"\n\t\"path/filepath\"\n\t\"regexp\"\n\t\"strings\"\n\n\t\"github.com/re-movery/re-movery/internal/core\"\n)\n\n// JavaScriptDetector is a detector for JavaScript code\ntype JavaScriptDetector struct {\n\tsignatures []core.Signature\n}\n\n// NewJavaScriptDetector creates a new JavaScript detector\nfunc NewJavaScriptDetector() *JavaScriptDetector {\n\tdetector := &JavaScriptDetector{}\n\tdetector.loadSignatures()\n\treturn detector\n}\n\n// Name returns the name of the detector\nfunc (d *JavaScriptDetector) Name() string {\n\treturn \"javascript\"\n}\n\n// SupportedLanguages returns the list of supported languages\nfunc (d *JavaScriptDetector) SupportedLanguages() []string {\n\treturn []string{\"javascript\", \"js\", \"jsx\", \"ts\", \"tsx\"}\n}\n\n// DetectFile detects vulnerabilities in a file\nfunc (d *JavaScriptDetector) DetectFile(filePath string) ([]core.Match, error) {\n\t// Check if file is a JavaScript file\n\text := filepath.Ext(filePath)\n\tif ext != \".js\" && ext != \".jsx\" && ext != \".ts\" && ext != \".tsx\" {\n\t\treturn nil, nil\n\t}\n\n\t// Read file\n\tcontent, err := ioutil.ReadFile(filePath)\n\tif err != nil {\n\t\treturn nil, err\n\t}\n\n\treturn d.DetectCode(string(content), filePath)\n}\n\n// DetectCode detects vulnerabilities in code\nfunc (d *JavaScriptDetector) DetectCode(code string, filePath string) ([]core.Match, error) {\n\tmatches := []core.Match{}\n\n\t// Scan code line by line\n\tscanner := bufio.NewScanner(strings.NewReader(code))\n\tlineNumber := 0\n\tfor scanner.Scan() {\n\t\tlineNumber++\n\t\tline := scanner.Text()\n\n\t\t// Check each signature\n\t\tfor _, signature := range d.signatures {\n\t\t\tfor _, pattern := range signature.CodePatterns {\n\t\t\t\tre, err := regexp.Compile(pattern)\n\t\t\t\tif err != nil {\n\t\t\t\t\tcontinue\n\t\t\t\t}\n\n\t\t\t\tif re.MatchString(line) {\n\t\t\t\t\tmatch := core.Match{\n\t\t\t\t\t\tSignature: signature,\n\t\t\t\t\t\tFilePath: filePath,\n\t\t\t\t\t\tLineNumber: lineNumber,\n\t\t\t\t\t\tMatchedCode: line,\n\t\t\t\t\t\tConfidence: d.calculateConfidence(line, pattern),\n\t\t\t\t\t}\n\t\t\t\t\tmatches = append(matches, match)\n\t\t\t\t}\n\t\t\t}\n\t\t}\n\t}\n\n\t// Perform additional JavaScript-specific checks\n\tmatches = append(matches, d.checkJavaScriptSpecificIssues(code, filePath)...)\n\n\treturn matches, nil\n}\n\n// loadSignatures loads the signatures for JavaScript code\nfunc (d *JavaScriptDetector) loadSignatures() {\n\td.signatures = []core.Signature{\n\t\t{\n\t\t\tID: \"JS001\",\n\t\t\tName: \"Dangerous eval() usage\",\n\t\t\tSeverity: \"high\",\n\t\t\tDescription: \"Using eval() can execute arbitrary code and is a security risk\",\n\t\t\tCodePatterns: []string{\n\t\t\t\t`eval\\s*\\([^)]*\\)`,\n\t\t\t},\n\t\t\tReferences: []string{\n\t\t\t\t\"https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/eval\",\n\t\t\t},\n\t\t},\n\t\t{\n\t\t\tID: \"JS002\",\n\t\t\tName: \"Dangerous Function() constructor\",\n\t\t\tSeverity: \"high\",\n\t\t\tDescription: \"Using Function() constructor can execute arbitrary code and is a security risk\",\n\t\t\tCodePatterns: []string{\n\t\t\t\t`new\\s+Function\\s*\\([^)]*\\)`,\n\t\t\t\t`Function\\s*\\([^)]*\\)`,\n\t\t\t},\n\t\t\tReferences: []string{\n\t\t\t\t\"https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Function\",\n\t\t\t},\n\t\t},\n\t\t{\n\t\t\tID: \"JS003\",\n\t\t\tName: \"DOM-based XSS risk\",\n\t\t\tSeverity: \"high\",\n\t\t\tDescription: \"Manipulating innerHTML with user input can lead to XSS\",\n\t\t\tCodePatterns: []string{\n\t\t\t\t`\\.innerHTML\\s*=`,\n\t\t\t\t`\\.outerHTML\\s*=`,\n\t\t\t\t`document\\.write\\s*\\(`,\n\t\t\t\t`document\\.writeln\\s*\\(`,\n\t\t\t},\n\t\t\tReferences: []string{\n\t\t\t\t\"https://owasp.org/www-community/attacks/xss/\",\n\t\t\t},\n\t\t},\n\t\t{\n\t\t\tID: \"JS004\",\n\t\t\tName: \"Insecure random number generation\",\n\t\t\tSeverity: \"medium\",\n\t\t\tDescription: \"Using Math.random() for security purposes is not recommended\",\n\t\t\tCodePatterns: []string{\n\t\t\t\t`Math\\.random\\s*\\(\\)`,\n\t\t\t},\n\t\t\tReferences: []string{\n\t\t\t\t\"https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Math/random\",\n\t\t\t},\n\t\t},\n\t\t{\n\t\t\tID: \"JS005\",\n\t\t\tName: \"Hardcoded credentials\",\n\t\t\tSeverity: \"high\",\n\t\t\tDescription: \"Hardcoded credentials are a security risk\",\n\t\t\tCodePatterns: []string{\n\t\t\t\t`password\\s*=\\s*['\\\"][^'\\\"]{3,}['\\\"]`,\n\t\t\t\t`passwd\\s*=\\s*['\\\"][^'\\\"]{3,}['\\\"]`,\n\t\t\t\t`pwd\\s*=\\s*['\\\"][^'\\\"]{3,}['\\\"]`,\n\t\t\t\t`secret\\s*=\\s*['\\\"][^'\\\"]{3,}['\\\"]`,\n\t\t\t\t`apiKey\\s*=\\s*['\\\"][^'\\\"]{3,}['\\\"]`,\n\t\t\t},\n\t\t\tReferences: []string{\n\t\t\t\t\"https://owasp.org/www-community/vulnerabilities/Use_of_hard-coded_credentials\",\n\t\t\t},\n\t\t},\n\t\t{\n\t\t\tID: \"JS006\",\n\t\t\tName: \"Insecure HTTP protocol\",\n\t\t\tSeverity: \"medium\",\n\t\t\tDescription: \"Using HTTP instead of HTTPS can expose data to eavesdropping\",\n\t\t\tCodePatterns: []string{\n\t\t\t\t`http:\\/\\/[^'\\\"]*['\\\"]`,\n\t\t\t},\n\t\t\tReferences: []string{\n\t\t\t\t\"https://owasp.org/www-project-top-ten/2017/A3_2017-Sensitive_Data_Exposure\",\n\t\t\t},\n\t\t},\n\t\t{\n\t\t\tID: \"JS007\",\n\t\t\tName: \"Potential prototype pollution\",\n\t\t\tSeverity: \"high\",\n\t\t\tDescription: \"Modifying Object.prototype can lead to prototype pollution vulnerabilities\",\n\t\t\tCodePatterns: []string{\n\t\t\t\t`Object\\.prototype\\.[^=]+=`,\n\t\t\t\t`__proto__\\.[^=]+=`,\n\t\t\t},\n\t\t\tReferences: []string{\n\t\t\t\t\"https://github.com/HoLyVieR/prototype-pollution-nsec18/blob/master/paper/JavaScript_prototype_pollution_attack_in_NodeJS.pdf\",\n\t\t\t},\n\t\t},\n\t\t{\n\t\t\tID: \"JS008\",\n\t\t\tName: \"Insecure JWT verification\",\n\t\t\tSeverity: \"high\",\n\t\t\tDescription: \"Not verifying JWT signatures can lead to authentication bypass\",\n\t\t\tCodePatterns: []string{\n\t\t\t\t`jwt\\.verify\\s*\\([^,]*,\\s*['\\\"]?none['\\\"]?[^)]*\\)`,\n\t\t\t},\n\t\t\tReferences: []string{\n\t\t\t\t\"https://auth0.com/blog/critical-vulnerabilities-in-json-web-token-libraries/\",\n\t\t\t},\n\t\t},\n\t\t{\n\t\t\tID: \"JS009\",\n\t\t\tName: \"Insecure cookie settings\",\n\t\t\tSeverity: \"medium\",\n\t\t\tDescription: \"Cookies without secure or httpOnly flags can be vulnerable to theft\",\n\t\t\tCodePatterns: []string{\n\t\t\t\t`document\\.cookie\\s*=\\s*[^;]*(?!secure|httpOnly)`,\n\t\t\t\t`\\.cookie\\s*\\([^)]*(?!secure|httpOnly)[^)]*\\)`,\n\t\t\t},\n\t\t\tReferences: []string{\n\t\t\t\t\"https://owasp.org/www-community/controls/SecureCookieAttribute\",\n\t\t\t},\n\t\t},\n\t\t{\n\t\t\tID: \"JS010\",\n\t\t\tName: \"Debug mode enabled\",\n\t\t\tSeverity: \"medium\",\n\t\t\tDescription: \"Running applications in debug mode can expose sensitive information\",\n\t\t\tCodePatterns: []string{\n\t\t\t\t`debug\\s*:\\s*true`,\n\t\t\t\t`debugMode\\s*=\\s*true`,\n\t\t\t},\n\t\t\tReferences: []string{\n\t\t\t\t\"https://expressjs.com/en/advanced/best-practice-security.html\",\n\t\t\t},\n\t\t},\n\t}\n}\n\n// calculateConfidence calculates the confidence of a match\nfunc (d *JavaScriptDetector) calculateConfidence(matchedCode string, pattern string) float64 {\n\t// Base confidence\n\tconfidence := 0.8\n\n\t// Adjust based on match length\n\tif len(matchedCode) > 10 {\n\t\tconfidence += 0.05\n\t}\n\n\t// Adjust based on context\n\tif strings.Contains(matchedCode, \"import\") || strings.Contains(matchedCode, \"require\") {\n\t\tconfidence += 0.05\n\t}\n\n\t// Adjust based on pattern specificity\n\tif len(pattern) > 20 {\n\t\tconfidence += 0.05\n\t}\n\n\t// Adjust based on function call parameters\n\tif strings.Contains(matchedCode, \"(\") && strings.Contains(matchedCode, \")\") {\n\t\tconfidence += 0.05\n\t}\n\n\t// Ensure confidence is between 0 and 1\n\tif confidence > 1.0 {\n\t\tconfidence = 1.0\n\t}\n\n\treturn confidence\n}\n\n// checkJavaScriptSpecificIssues performs additional JavaScript-specific checks\nfunc (d *JavaScriptDetector) checkJavaScriptSpecificIssues(code string, filePath string) []core.Match {\n\tmatches := []core.Match{}\n\n\t// Check for use of console.log in production code\n\tconsoleLogRe := regexp.MustCompile(`console\\.log\\s*\\(`)\n\tconsoleLogMatches := consoleLogRe.FindAllStringIndex(code, -1)\n\tfor _, match := range consoleLogMatches {\n\t\t// Count line number\n\t\tlineNumber := 1 + strings.Count(code[:match[0]], \"\\n\")\n\t\tmatchedCode := code[match[0]:match[1]] + \"...)\"\n\n\t\tmatches = append(matches, core.Match{\n\t\t\tSignature: core.Signature{\n\t\t\t\tID: \"JS011\",\n\t\t\t\tName: \"Console logging in production\",\n\t\t\t\tSeverity: \"low\",\n\t\t\t\tDescription: \"Console logging should be removed from production code\",\n\t\t\t\tCodePatterns: []string{\n\t\t\t\t\t`console\\.log\\s*\\(`,\n\t\t\t\t},\n\t\t\t},\n\t\t\tFilePath: filePath,\n\t\t\tLineNumber: lineNumber,\n\t\t\tMatchedCode: matchedCode,\n\t\t\tConfidence: 0.7,\n\t\t})\n\t}\n\n\t// Check for use of alert in production code\n\talertRe := regexp.MustCompile(`alert\\s*\\(`)\n\talertMatches := alertRe.FindAllStringIndex(code, -1)\n\tfor _, match := range alertMatches {\n\t\t// Count line number\n\t\tlineNumber := 1 + strings.Count(code[:match[0]], \"\\n\")\n\t\tmatchedCode := code[match[0]:match[1]] + \"...)\"\n\n\t\tmatches = append(matches, core.Match{\n\t\t\tSignature: core.Signature{\n\t\t\t\tID: \"JS012\",\n\t\t\t\tName: \"Alert in production\",\n\t\t\t\tSeverity: \"low\",\n\t\t\t\tDescription: \"Alert dialogs should be removed from production code\",\n\t\t\t\tCodePatterns: []string{\n\t\t\t\t\t`alert\\s*\\(`,\n\t\t\t\t},\n\t\t\t},\n\t\t\tFilePath: filePath,\n\t\t\tLineNumber: lineNumber,\n\t\t\tMatchedCode: matchedCode,\n\t\t\tConfidence: 0.7,\n\t\t})\n\t}\n\n\treturn matches\n} " }, { "path": "go/internal/detectors/python.go", "content": "package detectors\n\nimport (\n\t\"bufio\"\n\t\"io/ioutil\"\n\t\"os\"\n\t\"path/filepath\"\n\t\"regexp\"\n\t\"strings\"\n\n\t\"github.com/re-movery/re-movery/internal/core\"\n)\n\n// PythonDetector is a detector for Python code\ntype PythonDetector struct {\n\tsignatures []core.Signature\n}\n\n// NewPythonDetector creates a new Python detector\nfunc NewPythonDetector() *PythonDetector {\n\tdetector := &PythonDetector{}\n\tdetector.loadSignatures()\n\treturn detector\n}\n\n// Name returns the name of the detector\nfunc (d *PythonDetector) Name() string {\n\treturn \"python\"\n}\n\n// SupportedLanguages returns the list of supported languages\nfunc (d *PythonDetector) SupportedLanguages() []string {\n\treturn []string{\"python\", \"py\"}\n}\n\n// DetectFile detects vulnerabilities in a file\nfunc (d *PythonDetector) DetectFile(filePath string) ([]core.Match, error) {\n\t// Check if file is a Python file\n\tif filepath.Ext(filePath) != \".py\" {\n\t\treturn nil, nil\n\t}\n\n\t// Read file\n\tcontent, err := ioutil.ReadFile(filePath)\n\tif err != nil {\n\t\treturn nil, err\n\t}\n\n\treturn d.DetectCode(string(content), filePath)\n}\n\n// DetectCode detects vulnerabilities in code\nfunc (d *PythonDetector) DetectCode(code string, filePath string) ([]core.Match, error) {\n\tmatches := []core.Match{}\n\n\t// Scan code line by line\n\tscanner := bufio.NewScanner(strings.NewReader(code))\n\tlineNumber := 0\n\tfor scanner.Scan() {\n\t\tlineNumber++\n\t\tline := scanner.Text()\n\n\t\t// Check each signature\n\t\tfor _, signature := range d.signatures {\n\t\t\tfor _, pattern := range signature.CodePatterns {\n\t\t\t\tre, err := regexp.Compile(pattern)\n\t\t\t\tif err != nil {\n\t\t\t\t\tcontinue\n\t\t\t\t}\n\n\t\t\t\tif re.MatchString(line) {\n\t\t\t\t\tmatch := core.Match{\n\t\t\t\t\t\tSignature: signature,\n\t\t\t\t\t\tFilePath: filePath,\n\t\t\t\t\t\tLineNumber: lineNumber,\n\t\t\t\t\t\tMatchedCode: line,\n\t\t\t\t\t\tConfidence: d.calculateConfidence(line, pattern),\n\t\t\t\t\t}\n\t\t\t\t\tmatches = append(matches, match)\n\t\t\t\t}\n\t\t\t}\n\t\t}\n\t}\n\n\t// Perform additional Python-specific checks\n\tmatches = append(matches, d.checkPythonSpecificIssues(code, filePath)...)\n\n\treturn matches, nil\n}\n\n// loadSignatures loads the signatures for Python code\nfunc (d *PythonDetector) loadSignatures() {\n\td.signatures = []core.Signature{\n\t\t{\n\t\t\tID: \"PY001\",\n\t\t\tName: \"Dangerous eval() usage\",\n\t\t\tSeverity: \"high\",\n\t\t\tDescription: \"Using eval() can execute arbitrary code and is a security risk\",\n\t\t\tCodePatterns: []string{\n\t\t\t\t`eval\\s*\\([^)]*\\)`,\n\t\t\t},\n\t\t\tReferences: []string{\n\t\t\t\t\"https://docs.python.org/3/library/functions.html#eval\",\n\t\t\t},\n\t\t},\n\t\t{\n\t\t\tID: \"PY002\",\n\t\t\tName: \"Dangerous exec() usage\",\n\t\t\tSeverity: \"high\",\n\t\t\tDescription: \"Using exec() can execute arbitrary code and is a security risk\",\n\t\t\tCodePatterns: []string{\n\t\t\t\t`exec\\s*\\([^)]*\\)`,\n\t\t\t},\n\t\t\tReferences: []string{\n\t\t\t\t\"https://docs.python.org/3/library/functions.html#exec\",\n\t\t\t},\n\t\t},\n\t\t{\n\t\t\tID: \"PY003\",\n\t\t\tName: \"Insecure pickle usage\",\n\t\t\tSeverity: \"high\",\n\t\t\tDescription: \"Using pickle with untrusted data can lead to arbitrary code execution\",\n\t\t\tCodePatterns: []string{\n\t\t\t\t`pickle\\.loads\\s*\\([^)]*\\)`,\n\t\t\t\t`pickle\\.load\\s*\\([^)]*\\)`,\n\t\t\t},\n\t\t\tReferences: []string{\n\t\t\t\t\"https://docs.python.org/3/library/pickle.html\",\n\t\t\t},\n\t\t},\n\t\t{\n\t\t\tID: \"PY004\",\n\t\t\tName: \"SQL Injection risk\",\n\t\t\tSeverity: \"high\",\n\t\t\tDescription: \"String formatting in SQL queries can lead to SQL injection\",\n\t\t\tCodePatterns: []string{\n\t\t\t\t`execute\\s*\\(['\\\"][^'\\\"]*%[^'\\\"]*['\\\"]`,\n\t\t\t\t`execute\\s*\\(['\\\"][^'\\\"]*\\{\\s*[^}]*\\}[^'\\\"]*['\\\"]\\.format`,\n\t\t\t\t`execute\\s*\\(['\\\"][^'\\\"]*\\+[^'\\\"]*['\\\"]`,\n\t\t\t},\n\t\t\tReferences: []string{\n\t\t\t\t\"https://owasp.org/www-community/attacks/SQL_Injection\",\n\t\t\t},\n\t\t},\n\t\t{\n\t\t\tID: \"PY005\",\n\t\t\tName: \"Insecure random number generation\",\n\t\t\tSeverity: \"medium\",\n\t\t\tDescription: \"Using random module for security purposes is not recommended\",\n\t\t\tCodePatterns: []string{\n\t\t\t\t`random\\.(?:random|randint|choice|randrange)`,\n\t\t\t},\n\t\t\tReferences: []string{\n\t\t\t\t\"https://docs.python.org/3/library/random.html\",\n\t\t\t},\n\t\t},\n\t\t{\n\t\t\tID: \"PY006\",\n\t\t\tName: \"Hardcoded credentials\",\n\t\t\tSeverity: \"high\",\n\t\t\tDescription: \"Hardcoded credentials are a security risk\",\n\t\t\tCodePatterns: []string{\n\t\t\t\t`password\\s*=\\s*['\\\"][^'\\\"]{3,}['\\\"]`,\n\t\t\t\t`passwd\\s*=\\s*['\\\"][^'\\\"]{3,}['\\\"]`,\n\t\t\t\t`pwd\\s*=\\s*['\\\"][^'\\\"]{3,}['\\\"]`,\n\t\t\t\t`secret\\s*=\\s*['\\\"][^'\\\"]{3,}['\\\"]`,\n\t\t\t\t`api_key\\s*=\\s*['\\\"][^'\\\"]{3,}['\\\"]`,\n\t\t\t},\n\t\t\tReferences: []string{\n\t\t\t\t\"https://owasp.org/www-community/vulnerabilities/Use_of_hard-coded_credentials\",\n\t\t\t},\n\t\t},\n\t\t{\n\t\t\tID: \"PY007\",\n\t\t\tName: \"Insecure hash function\",\n\t\t\tSeverity: \"medium\",\n\t\t\tDescription: \"Using weak hash functions like MD5 or SHA1\",\n\t\t\tCodePatterns: []string{\n\t\t\t\t`hashlib\\.md5`,\n\t\t\t\t`hashlib\\.sha1`,\n\t\t\t},\n\t\t\tReferences: []string{\n\t\t\t\t\"https://owasp.org/www-community/vulnerabilities/Insufficient_entropy\",\n\t\t\t},\n\t\t},\n\t\t{\n\t\t\tID: \"PY008\",\n\t\t\tName: \"Temporary file creation risk\",\n\t\t\tSeverity: \"medium\",\n\t\t\tDescription: \"Insecure temporary file creation can lead to race conditions\",\n\t\t\tCodePatterns: []string{\n\t\t\t\t`open\\s*\\(['\\\"][^'\\\"]*\\/tmp[^'\\\"]*['\\\"]`,\n\t\t\t\t`tempfile\\.mktemp`,\n\t\t\t},\n\t\t\tReferences: []string{\n\t\t\t\t\"https://docs.python.org/3/library/tempfile.html\",\n\t\t\t},\n\t\t},\n\t\t{\n\t\t\tID: \"PY009\",\n\t\t\tName: \"Insecure deserialization\",\n\t\t\tSeverity: \"high\",\n\t\t\tDescription: \"Deserializing untrusted data can lead to arbitrary code execution\",\n\t\t\tCodePatterns: []string{\n\t\t\t\t`yaml\\.load\\s*\\([^)]*\\)`,\n\t\t\t\t`json\\.loads\\s*\\([^)]*\\)`,\n\t\t\t},\n\t\t\tReferences: []string{\n\t\t\t\t\"https://owasp.org/www-community/vulnerabilities/Deserialization_of_untrusted_data\",\n\t\t\t},\n\t\t},\n\t\t{\n\t\t\tID: \"PY010\",\n\t\t\tName: \"Debug mode enabled\",\n\t\t\tSeverity: \"medium\",\n\t\t\tDescription: \"Running applications in debug mode can expose sensitive information\",\n\t\t\tCodePatterns: []string{\n\t\t\t\t`debug\\s*=\\s*True`,\n\t\t\t\t`app\\.run\\s*\\([^)]*debug\\s*=\\s*True[^)]*\\)`,\n\t\t\t},\n\t\t\tReferences: []string{\n\t\t\t\t\"https://flask.palletsprojects.com/en/2.0.x/config/#DEBUG\",\n\t\t\t},\n\t\t},\n\t}\n}\n\n// calculateConfidence calculates the confidence of a match\nfunc (d *PythonDetector) calculateConfidence(matchedCode string, pattern string) float64 {\n\t// Base confidence\n\tconfidence := 0.8\n\n\t// Adjust based on match length\n\tif len(matchedCode) > 10 {\n\t\tconfidence += 0.05\n\t}\n\n\t// Adjust based on context\n\tif strings.Contains(matchedCode, \"import\") {\n\t\tconfidence += 0.05\n\t}\n\n\t// Adjust based on pattern specificity\n\tif len(pattern) > 20 {\n\t\tconfidence += 0.05\n\t}\n\n\t// Adjust based on function call parameters\n\tif strings.Contains(matchedCode, \"(\") && strings.Contains(matchedCode, \")\") {\n\t\tconfidence += 0.05\n\t}\n\n\t// Ensure confidence is between 0 and 1\n\tif confidence > 1.0 {\n\t\tconfidence = 1.0\n\t}\n\n\treturn confidence\n}\n\n// checkPythonSpecificIssues performs additional Python-specific checks\nfunc (d *PythonDetector) checkPythonSpecificIssues(code string, filePath string) []core.Match {\n\tmatches := []core.Match{}\n\n\t// Check for empty except blocks\n\temptyExceptRe := regexp.MustCompile(`(?m)^(\\s*)except(\\s+\\w+)?:\\s*$`)\n\temptyExceptMatches := emptyExceptRe.FindAllStringIndex(code, -1)\n\tfor _, match := range emptyExceptMatches {\n\t\t// Count line number\n\t\tlineNumber := 1 + strings.Count(code[:match[0]], \"\\n\")\n\t\tmatchedCode := code[match[0]:match[1]]\n\n\t\tmatches = append(matches, core.Match{\n\t\t\tSignature: core.Signature{\n\t\t\t\tID: \"PY011\",\n\t\t\t\tName: \"Empty except block\",\n\t\t\t\tSeverity: \"medium\",\n\t\t\t\tDescription: \"Empty except blocks can hide errors and make debugging difficult\",\n\t\t\t\tCodePatterns: []string{\n\t\t\t\t\t`except(\\s+\\w+)?:\\s*$`,\n\t\t\t\t},\n\t\t\t},\n\t\t\tFilePath: filePath,\n\t\t\tLineNumber: lineNumber,\n\t\t\tMatchedCode: matchedCode,\n\t\t\tConfidence: 0.85,\n\t\t})\n\t}\n\n\t// Check for bare except blocks\n\tbareExceptRe := regexp.MustCompile(`(?m)^(\\s*)except:\\s*`)\n\tbareExceptMatches := bareExceptRe.FindAllStringIndex(code, -1)\n\tfor _, match := range bareExceptMatches {\n\t\t// Count line number\n\t\tlineNumber := 1 + strings.Count(code[:match[0]], \"\\n\")\n\t\tmatchedCode := code[match[0]:match[1]]\n\n\t\tmatches = append(matches, core.Match{\n\t\t\tSignature: core.Signature{\n\t\t\t\tID: \"PY012\",\n\t\t\t\tName: \"Bare except block\",\n\t\t\t\tSeverity: \"medium\",\n\t\t\t\tDescription: \"Bare except blocks can catch unexpected exceptions and hide errors\",\n\t\t\t\tCodePatterns: []string{\n\t\t\t\t\t`except:\\s*`,\n\t\t\t\t},\n\t\t\t},\n\t\t\tFilePath: filePath,\n\t\t\tLineNumber: lineNumber,\n\t\t\tMatchedCode: matchedCode,\n\t\t\tConfidence: 0.9,\n\t\t})\n\t}\n\n\treturn matches\n} " }, { "path": "go/internal/detectors/tests/detector_test.go", "content": " " }, { "path": "go/internal/detectors/vulnerability.go", "content": "package detectors\n\nimport (\n\t\"encoding/json\"\n\t\"fmt\"\n\t\"io/ioutil\"\n\t\"os\"\n\t\"regexp\"\n\t\"strings\"\n\t\"sync\"\n\n\t\"github.com/dave/dst\"\n\t\"github.com/dave/dst/decorator\"\n)\n\n// Signature 表示漏洞签名\ntype Signature struct {\n\tID string `json:\"id\"`\n\tName string `json:\"name\"`\n\tSeverity string `json:\"severity\"`\n\tCodePatterns []string `json:\"code_patterns\"`\n}\n\n// Match 表示漏洞匹配结果\ntype Match struct {\n\tSignature Signature\n\tLineNumber int\n\tMatchedCode string\n\tConfidence float64\n}\n\n// VulnerabilityDetector 漏洞检测器\ntype VulnerabilityDetector struct {\n\tsignatures []Signature\n\tminConfidence float64\n\tmu sync.RWMutex\n}\n\n// NewVulnerabilityDetector 创建新的漏洞检测器\nfunc NewVulnerabilityDetector() *VulnerabilityDetector {\n\treturn &VulnerabilityDetector{\n\t\tminConfidence: 0.7,\n\t}\n}\n\n// LoadSignatures 从JSON文件加载漏洞签名\nfunc (d *VulnerabilityDetector) LoadSignatures(signatureFile string) error {\n\tdata, err := ioutil.ReadFile(signatureFile)\n\tif err != nil {\n\t\treturn fmt.Errorf(\"读取签名文件失败: %v\", err)\n\t}\n\n\tvar sigData struct {\n\t\tSignatures []Signature `json:\"signatures\"`\n\t}\n\n\tif err := json.Unmarshal(data, &sigData); err != nil {\n\t\treturn fmt.Errorf(\"解析签名文件失败: %v\", err)\n\t}\n\n\td.mu.Lock()\n\td.signatures = sigData.Signatures\n\td.mu.Unlock()\n\n\treturn nil\n}\n\n// DetectFile 检测文件中的漏洞\nfunc (d *VulnerabilityDetector) DetectFile(filePath string) ([]Match, error) {\n\tcontent, err := ioutil.ReadFile(filePath)\n\tif err != nil {\n\t\treturn nil, fmt.Errorf(\"读取文件失败: %v\", err)\n\t}\n\n\tmatches := make([]Match, 0)\n\td.mu.RLock()\n\tsignatures := d.signatures\n\td.mu.RUnlock()\n\n\t// 使用goroutine并行处理每个签名\n\tvar wg sync.WaitGroup\n\tmatchChan := make(chan Match)\n\tdone := make(chan bool)\n\n\t// 启动收集结果的goroutine\n\tgo func() {\n\t\tfor match := range matchChan {\n\t\t\tmatches = append(matches, match)\n\t\t}\n\t\tdone <- true\n\t}()\n\n\tfor _, sig := range signatures {\n\t\twg.Add(1)\n\t\tgo func(signature Signature) {\n\t\t\tdefer wg.Done()\n\t\t\tfor _, pattern := range signature.CodePatterns {\n\t\t\t\tre, err := regexp.Compile(pattern)\n\t\t\t\tif err != nil {\n\t\t\t\t\tcontinue\n\t\t\t\t}\n\n\t\t\t\t// 查找所有匹配\n\t\t\t\tfor _, match := range re.FindAllStringIndex(string(content), -1) {\n\t\t\t\t\tmatchedCode := string(content[match[0]:match[1]])\n\t\t\t\t\tconfidence := d.calculateConfidence(matchedCode, pattern)\n\n\t\t\t\t\tif confidence >= d.minConfidence {\n\t\t\t\t\t\t// 计算行号\n\t\t\t\t\t\tlineNumber := 1 + strings.Count(string(content[:match[0]]), \"\\n\")\n\t\t\t\t\t\tmatchChan <- Match{\n\t\t\t\t\t\t\tSignature: signature,\n\t\t\t\t\t\t\tLineNumber: lineNumber,\n\t\t\t\t\t\t\tMatchedCode: matchedCode,\n\t\t\t\t\t\t\tConfidence: confidence,\n\t\t\t\t\t\t}\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t}\n\t\t}(sig)\n\t}\n\n\t// 等待所有goroutine完成并关闭通道\n\tgo func() {\n\t\twg.Wait()\n\t\tclose(matchChan)\n\t}()\n\n\t<-done\n\treturn matches, nil\n}\n\n// AnalyzeAST 分析AST节点中的漏洞\nfunc (d *VulnerabilityDetector) AnalyzeAST(filePath string) ([]Match, error) {\n\tfset, node, err := decorator.ParseFile(filePath, nil)\n\tif err != nil {\n\t\treturn nil, fmt.Errorf(\"解析文件失败: %v\", err)\n\t}\n\n\tmatches := make([]Match, 0)\n\td.mu.RLock()\n\tsignatures := d.signatures\n\td.mu.RUnlock()\n\n\t// 遍历AST\n\tdst.Inspect(node, func(n dst.Node) bool {\n\t\tif call, ok := n.(*dst.CallExpr); ok {\n\t\t\tvar funcName string\n\t\t\tswitch fun := call.Fun.(type) {\n\t\t\tcase *dst.Ident:\n\t\t\t\tfuncName = fun.Name\n\t\t\tcase *dst.SelectorExpr:\n\t\t\t\tif x, ok := fun.X.(*dst.Ident); ok {\n\t\t\t\t\tfuncName = x.Name + \".\" + fun.Sel.Name\n\t\t\t\t}\n\t\t\tdefault:\n\t\t\t\treturn true\n\t\t\t}\n\n\t\t\t// 检查是否匹配任何签名\n\t\t\tfor _, sig := range signatures {\n\t\t\t\tfor _, pattern := range sig.CodePatterns {\n\t\t\t\t\tif matched, _ := regexp.MatchString(pattern, funcName); matched {\n\t\t\t\t\t\tmatches = append(matches, Match{\n\t\t\t\t\t\t\tSignature: sig,\n\t\t\t\t\t\t\tLineNumber: fset.Position(call.Pos()).Line,\n\t\t\t\t\t\t\tMatchedCode: funcName,\n\t\t\t\t\t\t\tConfidence: 0.9,\n\t\t\t\t\t\t})\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t}\n\t\t}\n\t\treturn true\n\t})\n\n\treturn matches, nil\n}\n\n// DetectSimilarPatterns 检测相似的漏洞模式\nfunc (d *VulnerabilityDetector) DetectSimilarPatterns(filePath string, threshold float64) ([]Match, error) {\n\tfset, node, err := decorator.ParseFile(filePath, nil)\n\tif err != nil {\n\t\treturn nil, fmt.Errorf(\"解析文件失败: %v\", err)\n\t}\n\n\tmatches := make([]Match, 0)\n\td.mu.RLock()\n\tsignatures := d.signatures\n\td.mu.RUnlock()\n\n\t// 遍历AST查找相似模式\n\tdst.Inspect(node, func(n dst.Node) bool {\n\t\tif call, ok := n.(*dst.CallExpr); ok {\n\t\t\tvar funcName string\n\t\t\tswitch fun := call.Fun.(type) {\n\t\t\tcase *dst.Ident:\n\t\t\t\tfuncName = fun.Name\n\t\t\tcase *dst.SelectorExpr:\n\t\t\t\tif x, ok := fun.X.(*dst.Ident); ok {\n\t\t\t\t\tfuncName = x.Name + \".\" + fun.Sel.Name\n\t\t\t\t}\n\t\t\tdefault:\n\t\t\t\treturn true\n\t\t\t}\n\n\t\t\t// 检查每个签名\n\t\t\tfor _, sig := range signatures {\n\t\t\t\tfor _, pattern := range sig.CodePatterns {\n\t\t\t\t\tsimilarity := d.calculateSimilarity(funcName, pattern)\n\t\t\t\t\tif similarity >= threshold {\n\t\t\t\t\t\tmatches = append(matches, Match{\n\t\t\t\t\t\t\tSignature: sig,\n\t\t\t\t\t\t\tLineNumber: fset.Position(call.Pos()).Line,\n\t\t\t\t\t\t\tMatchedCode: funcName,\n\t\t\t\t\t\t\tConfidence: similarity,\n\t\t\t\t\t\t})\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t}\n\t\t}\n\t\treturn true\n\t})\n\n\treturn matches, nil\n}\n\n// calculateConfidence 计算匹配的置信度\nfunc (d *VulnerabilityDetector) calculateConfidence(matchedCode, pattern string) float64 {\n\t// 基本匹配的置信度为0.7\n\tconfidence := 0.7\n\n\t// 根据匹配的完整性增加置信度\n\tif len(matchedCode) > 10 {\n\t\tconfidence += 0.1\n\t}\n\n\t// 根据上下文增加置信度\n\tif strings.Contains(matchedCode, \"import\") {\n\t\tconfidence += 0.1\n\t}\n\n\t// 根据模式的特异性增加置信度\n\tif len(pattern) > 20 {\n\t\tconfidence += 0.1\n\t}\n\n\tif confidence > 1.0 {\n\t\tconfidence = 1.0\n\t}\n\treturn confidence\n}\n\n// calculateSimilarity 计算两个字符串的相似度\nfunc (d *VulnerabilityDetector) calculateSimilarity(str1, str2 string) float64 {\n\t// 使用最长公共子序列(LCS)计算相似度\n\tm, n := len(str1), len(str2)\n\tdp := make([][]int, m+1)\n\tfor i := range dp {\n\t\tdp[i] = make([]int, n+1)\n\t}\n\n\tfor i := 1; i <= m; i++ {\n\t\tfor j := 1; j <= n; j++ {\n\t\t\tif str1[i-1] == str2[j-1] {\n\t\t\t\tdp[i][j] = dp[i-1][j-1] + 1\n\t\t\t} else {\n\t\t\t\tdp[i][j] = max(dp[i-1][j], dp[i][j-1])\n\t\t\t}\n\t\t}\n\t}\n\n\tlcsLength := dp[m][n]\n\tmaxLen := max(m, n)\n\tif maxLen == 0 {\n\t\treturn 0\n\t}\n\treturn float64(lcsLength) / float64(maxLen)\n}\n\n// max 返回两个整数中的较大值\nfunc max(a, b int) int {\n\tif a > b {\n\t\treturn a\n\t}\n\treturn b\n} " }, { "path": "go/internal/reporters/html.go", "content": "package reporters\n\nimport (\n\t\"fmt\"\n\t\"html/template\"\n\t\"os\"\n\t\"path/filepath\"\n\t\"sort\"\n\t\"time\"\n\n\t\"github.com/re-movery/re-movery/internal/core\"\n)\n\n// HTMLReporter is a reporter that generates HTML reports\ntype HTMLReporter struct{}\n\n// NewHTMLReporter creates a new HTML reporter\nfunc NewHTMLReporter() *HTMLReporter {\n\treturn &HTMLReporter{}\n}\n\n// GenerateReport generates a report\nfunc (r *HTMLReporter) GenerateReport(data core.ReportData, outputPath string) error {\n\t// Create output directory if it doesn't exist\n\toutputDir := filepath.Dir(outputPath)\n\tif err := os.MkdirAll(outputDir, 0755); err != nil {\n\t\treturn err\n\t}\n\n\t// Create output file\n\tfile, err := os.Create(outputPath)\n\tif err != nil {\n\t\treturn err\n\t}\n\tdefer file.Close()\n\n\t// Process data for the template\n\tprocessedData := r.processData(data)\n\n\t// Parse template\n\ttmpl, err := template.New(\"report\").Funcs(template.FuncMap{\n\t\t\"mul\": func(a, b float64) float64 {\n\t\t\treturn a * b\n\t\t},\n\t}).Parse(htmlTemplate)\n\tif err != nil {\n\t\treturn err\n\t}\n\n\t// Execute template\n\tif err := tmpl.Execute(file, processedData); err != nil {\n\t\treturn err\n\t}\n\n\treturn nil\n}\n\n// processData processes the report data for the template\nfunc (r *HTMLReporter) processData(data core.ReportData) map[string]interface{} {\n\t// Count vulnerabilities by type\n\tvulnCounts := make(map[string]int)\n\tfor _, matches := range data.Results {\n\t\tfor _, match := range matches {\n\t\t\tvulnCounts[match.Signature.Name]++\n\t\t}\n\t}\n\n\t// Sort vulnerabilities by count\n\ttype vulnCount struct {\n\t\tName string\n\t\tCount int\n\t}\n\tvulnCountList := []vulnCount{}\n\tfor name, count := range vulnCounts {\n\t\tvulnCountList = append(vulnCountList, vulnCount{Name: name, Count: count})\n\t}\n\tsort.Slice(vulnCountList, func(i, j int) bool {\n\t\treturn vulnCountList[i].Count > vulnCountList[j].Count\n\t})\n\n\t// Get top 10 vulnerabilities\n\ttopVulns := vulnCountList\n\tif len(topVulns) > 10 {\n\t\ttopVulns = topVulns[:10]\n\t}\n\n\t// Prepare data for the template\n\tprocessedData := map[string]interface{}{\n\t\t\"Title\": data.Title,\n\t\t\"Timestamp\": data.Timestamp,\n\t\t\"Results\": data.Results,\n\t\t\"Summary\": data.Summary,\n\t\t\"TopVulnerabilities\": map[string]interface{}{\n\t\t\t\"Labels\": func() []string {\n\t\t\t\tlabels := []string{}\n\t\t\t\tfor _, v := range topVulns {\n\t\t\t\t\tlabels = append(labels, v.Name)\n\t\t\t\t}\n\t\t\t\treturn labels\n\t\t\t}(),\n\t\t\t\"Data\": func() []int {\n\t\t\t\tcounts := []int{}\n\t\t\t\tfor _, v := range topVulns {\n\t\t\t\t\tcounts = append(counts, v.Count)\n\t\t\t\t}\n\t\t\t\treturn counts\n\t\t\t}(),\n\t\t},\n\t}\n\n\treturn processedData\n}\n\n// htmlTemplate is the HTML template for the report\nconst htmlTemplate = `\n\n\n \n \n {{ .Title }}\n \n \n\n\n

{{ .Title }}

\n \n
\n

Summary

\n
\n

{{ .Summary.High }}

\n

High Severity

\n
\n
\n

{{ .Summary.Medium }}

\n

Medium Severity

\n
\n
\n

{{ .Summary.Low }}

\n

Low Severity

\n
\n
\n

{{ .Summary.TotalFiles }}

\n

Files Scanned

\n
\n
\n \n
\n \n
\n \n

Top Vulnerabilities

\n
\n \n
\n \n

Detailed Results

\n {{range $file, $matches := .Results}}\n
\n
\n

{{$file}}

\n {{len $matches}} issues found\n
\n
\n \n \n \n \n \n \n \n \n \n \n {{range $match := $matches}}\n \n \n \n \n \n \n {{end}}\n \n
LineSeverityIssueConfidence
{{$match.LineNumber}}{{$match.Signature.Severity}}\n {{$match.Signature.Name}}\n

{{$match.Signature.Description}}

\n
{{$match.MatchedCode}}
\n 		{{printf \"%.0f%%\" (mul $match.Confidence 100)}}
\n
\n
\n {{end}}\n \n
\n

Report generated by Re-movery on {{.Timestamp}}

\n
\n \n \n\n` " }, { "path": "go/internal/reporters/json.go", "content": "package reporters\n\nimport (\n\t\"encoding/json\"\n\t\"os\"\n\t\"path/filepath\"\n\n\t\"github.com/re-movery/re-movery/internal/core\"\n)\n\n// JSONReporter is a reporter that generates JSON reports\ntype JSONReporter struct{}\n\n// NewJSONReporter creates a new JSON reporter\nfunc NewJSONReporter() *JSONReporter {\n\treturn &JSONReporter{}\n}\n\n// GenerateReport generates a report\nfunc (r *JSONReporter) GenerateReport(data core.ReportData, outputPath string) error {\n\t// Create output directory if it doesn't exist\n\toutputDir := filepath.Dir(outputPath)\n\tif err := os.MkdirAll(outputDir, 0755); err != nil {\n\t\treturn err\n\t}\n\n\t// Create output file\n\tfile, err := os.Create(outputPath)\n\tif err != nil {\n\t\treturn err\n\t}\n\tdefer file.Close()\n\n\t// Marshal data to JSON\n\tencoder := json.NewEncoder(file)\n\tencoder.SetIndent(\"\", \" \")\n\tif err := encoder.Encode(data); err != nil {\n\t\treturn err\n\t}\n\n\treturn nil\n} " }, { "path": "go/internal/reporters/xml.go", "content": "package reporters\n\nimport (\n\t\"encoding/xml\"\n\t\"os\"\n\t\"path/filepath\"\n\n\t\"github.com/re-movery/re-movery/internal/core\"\n)\n\n// XMLReporter is a reporter that generates XML reports\ntype XMLReporter struct{}\n\n// NewXMLReporter creates a new XML reporter\nfunc NewXMLReporter() *XMLReporter {\n\treturn &XMLReporter{}\n}\n\n// XMLReportData is the XML representation of the report data\ntype XMLReportData struct {\n\tXMLName xml.Name `xml:\"report\"`\n\tTitle string `xml:\"title\"`\n\tTimestamp string `xml:\"timestamp\"`\n\tSummary XMLSummary `xml:\"summary\"`\n\tResults []XMLFileResult `xml:\"results>file\"`\n}\n\n// XMLSummary is the XML representation of the summary\ntype XMLSummary struct {\n\tTotalFiles int `xml:\"totalFiles,attr\"`\n\tHigh int `xml:\"high,attr\"`\n\tMedium int `xml:\"medium,attr\"`\n\tLow int `xml:\"low,attr\"`\n}\n\n// XMLFileResult is the XML representation of a file result\ntype XMLFileResult struct {\n\tPath string `xml:\"path,attr\"`\n\tMatches []XMLMatch `xml:\"match\"`\n}\n\n// XMLMatch is the XML representation of a match\ntype XMLMatch struct {\n\tID string `xml:\"id,attr\"`\n\tName string `xml:\"name\"`\n\tSeverity string `xml:\"severity\"`\n\tDescription string `xml:\"description\"`\n\tLineNumber int `xml:\"lineNumber\"`\n\tMatchedCode string `xml:\"matchedCode\"`\n\tConfidence float64 `xml:\"confidence\"`\n}\n\n// GenerateReport generates a report\nfunc (r *XMLReporter) GenerateReport(data core.ReportData, outputPath string) error {\n\t// Create output directory if it doesn't exist\n\toutputDir := filepath.Dir(outputPath)\n\tif err := os.MkdirAll(outputDir, 0755); err != nil {\n\t\treturn err\n\t}\n\n\t// Create output file\n\tfile, err := os.Create(outputPath)\n\tif err != nil {\n\t\treturn err\n\t}\n\tdefer file.Close()\n\n\t// Convert data to XML format\n\txmlData := r.convertToXML(data)\n\n\t// Write XML header\n\tfile.WriteString(xml.Header)\n\n\t// Marshal data to XML\n\tencoder := xml.NewEncoder(file)\n\tencoder.Indent(\"\", \" \")\n\tif err := encoder.Encode(xmlData); err != nil {\n\t\treturn err\n\t}\n\n\treturn nil\n}\n\n// convertToXML converts the report data to XML format\nfunc (r *XMLReporter) convertToXML(data core.ReportData) XMLReportData {\n\txmlData := XMLReportData{\n\t\tTitle: data.Title,\n\t\tTimestamp: data.Timestamp,\n\t\tSummary: XMLSummary{\n\t\t\tTotalFiles: data.Summary.TotalFiles,\n\t\t\tHigh: data.Summary.High,\n\t\t\tMedium: data.Summary.Medium,\n\t\t\tLow: data.Summary.Low,\n\t\t},\n\t\tResults: []XMLFileResult{},\n\t}\n\n\t// Convert results\n\tfor filePath, matches := range data.Results {\n\t\tfileResult := XMLFileResult{\n\t\t\tPath: filePath,\n\t\t\tMatches: []XMLMatch{},\n\t\t}\n\n\t\tfor _, match := range matches {\n\t\t\txmlMatch := XMLMatch{\n\t\t\t\tID: match.Signature.ID,\n\t\t\t\tName: match.Signature.Name,\n\t\t\t\tSeverity: match.Signature.Severity,\n\t\t\t\tDescription: match.Signature.Description,\n\t\t\t\tLineNumber: match.LineNumber,\n\t\t\t\tMatchedCode: match.MatchedCode,\n\t\t\t\tConfidence: match.Confidence,\n\t\t\t}\n\t\t\tfileResult.Matches = append(fileResult.Matches, xmlMatch)\n\t\t}\n\n\t\txmlData.Results = append(xmlData.Results, fileResult)\n\t}\n\n\treturn xmlData\n} " }, { "path": "go/internal/utils/logging.go", "content": "package utils\n\nimport (\n \"io\"\n \"os\"\n \"sync\"\n\n \"github.com/sirupsen/logrus\"\n)\n\nvar (\n logger *logrus.Logger\n once sync.Once\n)\n\n// GetLogger returns the singleton logger instance\nfunc GetLogger() *logrus.Logger {\n once.Do(func() {\n logger = logrus.New()\n logger.SetFormatter(&logrus.TextFormatter{\n FullTimestamp: true,\n })\n logger.SetOutput(os.Stdout)\n logger.SetLevel(logrus.InfoLevel)\n })\n return logger\n}\n\n// FileLogger represents a logger that writes to a file\ntype FileLogger struct {\n *logrus.Logger\n file *os.File\n}\n\n// NewFileLogger creates a new file logger\nfunc NewFileLogger(filename string) (*FileLogger, error) {\n file, err := os.OpenFile(filename, os.O_CREATE|os.O_WRONLY|os.O_APPEND, 0666)\n if err != nil {\n return nil, err\n }\n\n logger := logrus.New()\n logger.SetFormatter(&logrus.JSONFormatter{})\n logger.SetOutput(io.MultiWriter(file, os.Stdout))\n\n return &FileLogger{\n Logger: logger,\n file: file,\n }, nil\n}\n\n// Close closes the log file\nfunc (fl *FileLogger) Close() error {\n if fl.file != nil {\n return fl.file.Close()\n }\n return nil\n}\n\n// SetVerbosity sets the logging level based on verbosity\nfunc SetVerbosity(verbose bool) {\n if verbose {\n GetLogger().SetLevel(logrus.DebugLevel)\n } else {\n GetLogger().SetLevel(logrus.InfoLevel)\n }\n} " }, { "path": "go/internal/utils/memory.go", "content": "package utils\n\nimport (\n \"container/list\"\n \"runtime\"\n \"sync\"\n \"time\"\n\n \"github.com/shirou/gopsutil/v3/mem\"\n)\n\n// MemoryMonitor monitors system memory usage\ntype MemoryMonitor struct {\n maxMemoryGB float64\n interval time.Duration\n stopChan chan struct{}\n}\n\n// NewMemoryMonitor creates a new memory monitor\nfunc NewMemoryMonitor(maxMemoryGB float64, interval time.Duration) *MemoryMonitor {\n return &MemoryMonitor{\n maxMemoryGB: maxMemoryGB,\n interval: interval,\n stopChan: make(chan struct{}),\n }\n}\n\n// Start starts monitoring memory usage\nfunc (mm *MemoryMonitor) Start() {\n go func() {\n ticker := time.NewTicker(mm.interval)\n defer ticker.Stop()\n\n for {\n select {\n case <-ticker.C:\n v, err := mem.VirtualMemory()\n if err != nil {\n GetLogger().Errorf(\"Failed to get memory stats: %v\", err)\n continue\n }\n\n usedGB := float64(v.Used) / (1024 * 1024 * 1024)\n if usedGB > mm.maxMemoryGB {\n GetLogger().Warnf(\"Memory usage (%.2f GB) exceeds limit (%.2f GB), triggering GC\", usedGB, mm.maxMemoryGB)\n runtime.GC()\n }\n case <-mm.stopChan:\n return\n }\n }\n }()\n}\n\n// Stop stops the memory monitor\nfunc (mm *MemoryMonitor) Stop() {\n close(mm.stopChan)\n}\n\n// LRUCache implements a thread-safe LRU cache\ntype LRUCache struct {\n capacity int\n cache map[interface{}]*list.Element\n ll *list.List\n mutex sync.RWMutex\n}\n\ntype entry struct {\n key interface{}\n value interface{}\n}\n\n// NewLRUCache creates a new LRU cache with the specified capacity\nfunc NewLRUCache(capacity int) *LRUCache {\n return &LRUCache{\n capacity: capacity,\n cache: make(map[interface{}]*list.Element),\n ll: list.New(),\n }\n}\n\n// Get retrieves a value from the cache\nfunc (c *LRUCache) Get(key interface{}) (interface{}, bool) {\n c.mutex.RLock()\n defer c.mutex.RUnlock()\n\n if elem, ok := c.cache[key]; ok {\n c.ll.MoveToFront(elem)\n return elem.Value.(*entry).value, true\n }\n return nil, false\n}\n\n// Put adds a value to the cache\nfunc (c *LRUCache) Put(key, value interface{}) {\n c.mutex.Lock()\n defer c.mutex.Unlock()\n\n if elem, ok := c.cache[key]; ok {\n c.ll.MoveToFront(elem)\n elem.Value.(*entry).value = value\n return\n }\n\n if c.ll.Len() >= c.capacity {\n oldest := c.ll.Back()\n if oldest != nil {\n c.ll.Remove(oldest)\n delete(c.cache, oldest.Value.(*entry).key)\n }\n }\n\n elem := c.ll.PushFront(&entry{key, value})\n c.cache[key] = elem\n} " }, { "path": "go/internal/utils/parallel.go", "content": "package utils\n\nimport (\n \"sync\"\n)\n\n// Job represents a unit of work\ntype Job interface {\n Execute() error\n}\n\n// WorkerPool manages a pool of workers for parallel processing\ntype WorkerPool struct {\n numWorkers int\n jobs chan Job\n results chan error\n wg sync.WaitGroup\n stopChan chan struct{}\n}\n\n// NewWorkerPool creates a new worker pool\nfunc NewWorkerPool(numWorkers int, queueSize int) *WorkerPool {\n return &WorkerPool{\n numWorkers: numWorkers,\n jobs: make(chan Job, queueSize),\n results: make(chan error, queueSize),\n stopChan: make(chan struct{}),\n }\n}\n\n// Start starts the worker pool\nfunc (wp *WorkerPool) Start() {\n for i := 0; i < wp.numWorkers; i++ {\n wp.wg.Add(1)\n go wp.worker()\n }\n}\n\n// worker processes jobs from the job queue\nfunc (wp *WorkerPool) worker() {\n defer wp.wg.Done()\n\n for {\n select {\n case job := <-wp.jobs:\n if job == nil {\n return\n }\n err := job.Execute()\n wp.results <- err\n case <-wp.stopChan:\n return\n }\n }\n}\n\n// Submit submits a job to the worker pool\nfunc (wp *WorkerPool) Submit(job Job) {\n wp.jobs <- job\n}\n\n// Stop stops the worker pool\nfunc (wp *WorkerPool) Stop() {\n close(wp.stopChan)\n wp.wg.Wait()\n close(wp.jobs)\n close(wp.results)\n}\n\n// Results returns the results channel\nfunc (wp *WorkerPool) Results() <-chan error {\n return wp.results\n} " }, { "path": "go/internal/utils/security.go", "content": "package utils\n\nimport (\n\t\"fmt\"\n\t\"go/ast\"\n\t\"go/parser\"\n\t\"go/token\"\n\t\"io/ioutil\"\n\t\"os\"\n\t\"regexp\"\n\t\"runtime\"\n\t\"strings\"\n\t\"sync\"\n\t\"time\"\n)\n\n// SecurityChecker 安全检查器\ntype SecurityChecker struct {\n\tsensitivePatterns map[string][]string\n\tmu sync.RWMutex\n}\n\n// NewSecurityChecker 创建新的安全检查器\nfunc NewSecurityChecker() *SecurityChecker {\n\treturn &SecurityChecker{\n\t\tsensitivePatterns: map[string][]string{\n\t\t\t\"file_access\": {\n\t\t\t\t`os\\.(Open|Create|Remove|RemoveAll|Chmod|Chown)`,\n\t\t\t\t`ioutil\\.(ReadFile|WriteFile)`,\n\t\t\t},\n\t\t\t\"network_access\": {\n\t\t\t\t`net\\.(Dial|Listen)`,\n\t\t\t\t`http\\.(Get|Post|Put|Delete)`,\n\t\t\t},\n\t\t\t\"code_execution\": {\n\t\t\t\t`exec\\.(Command|Run)`,\n\t\t\t\t`syscall\\.(Exec|StartProcess)`,\n\t\t\t},\n\t\t\t\"input_validation\": {\n\t\t\t\t`fmt\\.(Scan|Scanf|Scanln)`,\n\t\t\t\t`bufio\\.NewScanner`,\n\t\t\t},\n\t\t\t\"random_generation\": {\n\t\t\t\t`math/rand\\.(Int|Float|Perm)`,\n\t\t\t\t`crypto/rand\\.(Read|Prime)`,\n\t\t\t},\n\t\t\t\"sensitive_data\": {\n\t\t\t\t`(?i)(password|secret|key|token|credential)`,\n\t\t\t\t`fmt\\.Printf.*password`,\n\t\t\t},\n\t\t},\n\t}\n}\n\n// CheckMemoryUsage 检查内存使用情况\nfunc (c *SecurityChecker) CheckMemoryUsage(filePath string) (uint64, error) {\n\tvar m runtime.MemStats\n\truntime.ReadMemStats(&m)\n\tinitialAlloc := m.Alloc\n\n\t// 读取并执行文件\n\tcontent, err := ioutil.ReadFile(filePath)\n\tif err != nil {\n\t\treturn 0, fmt.Errorf(\"读取文件失败: %v\", err)\n\t}\n\n\t// 解析文件以检查内存使用\n\tfset := token.NewFileSet()\n\t_, err = parser.ParseFile(fset, filePath, content, parser.AllErrors)\n\tif err != nil {\n\t\treturn 0, fmt.Errorf(\"解析文件失败: %v\", err)\n\t}\n\n\truntime.ReadMemStats(&m)\n\tfinalAlloc := m.Alloc\n\n\treturn finalAlloc - initialAlloc, nil\n}\n\n// CheckExecutionTime 检查执行时间\nfunc (c *SecurityChecker) CheckExecutionTime(filePath string, timeout time.Duration) error {\n\tdone := make(chan bool)\n\tvar execErr error\n\n\tgo func() {\n\t\t// 读取并解析文件\n\t\tcontent, err := ioutil.ReadFile(filePath)\n\t\tif err != nil {\n\t\t\texecErr = fmt.Errorf(\"读取文件失败: %v\", err)\n\t\t\tdone <- true\n\t\t\treturn\n\t\t}\n\n\t\tfset := token.NewFileSet()\n\t\t_, err = parser.ParseFile(fset, filePath, content, parser.AllErrors)\n\t\tif err != nil {\n\t\t\texecErr = fmt.Errorf(\"解析文件失败: %v\", err)\n\t\t\tdone <- true\n\t\t\treturn\n\t\t}\n\n\t\tdone <- true\n\t}()\n\n\tselect {\n\tcase <-done:\n\t\treturn execErr\n\tcase <-time.After(timeout):\n\t\treturn fmt.Errorf(\"执行超时(>%v)\", timeout)\n\t}\n}\n\n// CheckFileAccess 检查文件访问安全性\nfunc (c *SecurityChecker) CheckFileAccess(filePath string) ([]string, error) {\n\tviolations := make([]string, 0)\n\tcontent, err := ioutil.ReadFile(filePath)\n\tif err != nil {\n\t\treturn nil, fmt.Errorf(\"读取文件失败: %v\", err)\n\t}\n\n\tc.mu.RLock()\n\tpatterns := c.sensitivePatterns[\"file_access\"]\n\tc.mu.RUnlock()\n\n\tfor _, pattern := range patterns {\n\t\tre, err := regexp.Compile(pattern)\n\t\tif err != nil {\n\t\t\tcontinue\n\t\t}\n\n\t\tmatches := re.FindAllString(string(content), -1)\n\t\tfor _, match := range matches {\n\t\t\tviolations = append(violations, fmt.Sprintf(\"发现敏感文件操作: %s\", match))\n\t\t}\n\t}\n\n\treturn violations, nil\n}\n\n// CheckNetworkAccess 检查网络访问安全性\nfunc (c *SecurityChecker) CheckNetworkAccess(filePath string) ([]string, error) {\n\tviolations := make([]string, 0)\n\tcontent, err := ioutil.ReadFile(filePath)\n\tif err != nil {\n\t\treturn nil, fmt.Errorf(\"读取文件失败: %v\", err)\n\t}\n\n\tc.mu.RLock()\n\tpatterns := c.sensitivePatterns[\"network_access\"]\n\tc.mu.RUnlock()\n\n\tfor _, pattern := range patterns {\n\t\tre, err := regexp.Compile(pattern)\n\t\tif err != nil {\n\t\t\tcontinue\n\t\t}\n\n\t\tmatches := re.FindAllString(string(content), -1)\n\t\tfor _, match := range matches {\n\t\t\tviolations = append(violations, fmt.Sprintf(\"发现敏感网络操作: %s\", match))\n\t\t}\n\t}\n\n\treturn violations, nil\n}\n\n// CheckInputValidation 检查输入验证\nfunc (c *SecurityChecker) CheckInputValidation(filePath string) ([]string, error) {\n\tissues := make([]string, 0)\n\tcontent, err := ioutil.ReadFile(filePath)\n\tif err != nil {\n\t\treturn nil, fmt.Errorf(\"读取文件失败: %v\", err)\n\t}\n\n\tfset := token.NewFileSet()\n\tfile, err := parser.ParseFile(fset, filePath, content, parser.AllErrors)\n\tif err != nil {\n\t\treturn nil, fmt.Errorf(\"解析文件失败: %v\", err)\n\t}\n\n\tast.Inspect(file, func(n ast.Node) bool {\n\t\tif call, ok := n.(*ast.CallExpr); ok {\n\t\t\tif sel, ok := call.Fun.(*ast.SelectorExpr); ok {\n\t\t\t\tif x, ok := sel.X.(*ast.Ident); ok {\n\t\t\t\t\tfuncName := x.Name + \".\" + sel.Sel.Name\n\t\t\t\t\tif strings.Contains(funcName, \"fmt.Scan\") || strings.Contains(funcName, \"bufio.NewScanner\") {\n\t\t\t\t\t\tissues = append(issues, fmt.Sprintf(\"未验证的输入: %s\", funcName))\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t}\n\t\t}\n\t\treturn true\n\t})\n\n\treturn issues, nil\n}\n\n// CheckRandomGeneration 检查随机数生成安全性\nfunc (c *SecurityChecker) CheckRandomGeneration(filePath string) ([]string, error) {\n\tissues := make([]string, 0)\n\tcontent, err := ioutil.ReadFile(filePath)\n\tif err != nil {\n\t\treturn nil, fmt.Errorf(\"读取文件失败: %v\", err)\n\t}\n\n\tc.mu.RLock()\n\tpatterns := c.sensitivePatterns[\"random_generation\"]\n\tc.mu.RUnlock()\n\n\tfor _, pattern := range patterns {\n\t\tre, err := regexp.Compile(pattern)\n\t\tif err != nil {\n\t\t\tcontinue\n\t\t}\n\n\t\tmatches := re.FindAllString(string(content), -1)\n\t\tfor _, match := range matches {\n\t\t\tif !strings.Contains(match, \"crypto/rand\") {\n\t\t\t\tissues = append(issues, fmt.Sprintf(\"不安全的随机数生成: %s\", match))\n\t\t\t}\n\t\t}\n\t}\n\n\treturn issues, nil\n}\n\n// CheckSensitiveData 检查敏感数据处理\nfunc (c *SecurityChecker) CheckSensitiveData(filePath string) ([]string, error) {\n\tissues := make([]string, 0)\n\tcontent, err := ioutil.ReadFile(filePath)\n\tif err != nil {\n\t\treturn nil, fmt.Errorf(\"读取文件失败: %v\", err)\n\t}\n\n\tc.mu.RLock()\n\tpatterns := c.sensitivePatterns[\"sensitive_data\"]\n\tc.mu.RUnlock()\n\n\tfor _, pattern := range patterns {\n\t\tre, err := regexp.Compile(pattern)\n\t\tif err != nil {\n\t\t\tcontinue\n\t\t}\n\n\t\tmatches := re.FindAllString(string(content), -1)\n\t\tfor _, match := range matches {\n\t\t\tissues = append(issues, fmt.Sprintf(\"敏感数据泄露风险: %s\", match))\n\t\t}\n\t}\n\n\treturn issues, nil\n}\n\n// CheckSandboxEscape 检查沙箱逃逸\nfunc (c *SecurityChecker) CheckSandboxEscape(filePath string) ([]string, error) {\n\tviolations := make([]string, 0)\n\tcontent, err := ioutil.ReadFile(filePath)\n\tif err != nil {\n\t\treturn nil, fmt.Errorf(\"读取文件失败: %v\", err)\n\t}\n\n\tfset := token.NewFileSet()\n\tfile, err := parser.ParseFile(fset, filePath, content, parser.AllErrors)\n\tif err != nil {\n\t\treturn nil, fmt.Errorf(\"解析文件失败: %v\", err)\n\t}\n\n\tast.Inspect(file, func(n ast.Node) bool {\n\t\tif call, ok := n.(*ast.CallExpr); ok {\n\t\t\tif sel, ok := call.Fun.(*ast.SelectorExpr); ok {\n\t\t\t\tif x, ok := sel.X.(*ast.Ident); ok {\n\t\t\t\t\tfuncName := x.Name + \".\" + sel.Sel.Name\n\t\t\t\t\tif strings.Contains(funcName, \"os.\") || strings.Contains(funcName, \"exec.\") {\n\t\t\t\t\t\tviolations = append(violations, fmt.Sprintf(\"危险的系统调用: %s\", funcName))\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t}\n\t\t}\n\t\treturn true\n\t})\n\n\treturn violations, nil\n}\n\n// PerformFullCheck 执行完整的安全检查\nfunc (c *SecurityChecker) PerformFullCheck(filePath string) (map[string]interface{}, error) {\n\tresults := make(map[string]interface{})\n\n\t// 检查内存使用\n\tmemoryUsage, err := c.CheckMemoryUsage(filePath)\n\tif err != nil {\n\t\tresults[\"memory_usage\"] = err.Error()\n\t} else {\n\t\tresults[\"memory_usage\"] = memoryUsage\n\t}\n\n\t// 检查执行时间\n\terr = c.CheckExecutionTime(filePath, 5*time.Second)\n\tif err != nil {\n\t\tresults[\"execution_time\"] = err.Error()\n\t} else {\n\t\tresults[\"execution_time\"] = \"OK\"\n\t}\n\n\t// 检查文件访问\n\tfileAccess, err := c.CheckFileAccess(filePath)\n\tif err != nil {\n\t\tresults[\"file_access\"] = err.Error()\n\t} else {\n\t\tresults[\"file_access\"] = fileAccess\n\t}\n\n\t// 检查网络访问\n\tnetworkAccess, err := c.CheckNetworkAccess(filePath)\n\tif err != nil {\n\t\tresults[\"network_access\"] = err.Error()\n\t} else {\n\t\tresults[\"network_access\"] = networkAccess\n\t}\n\n\t// 检查输入验证\n\tinputValidation, err := c.CheckInputValidation(filePath)\n\tif err != nil {\n\t\tresults[\"input_validation\"] = err.Error()\n\t} else {\n\t\tresults[\"input_validation\"] = inputValidation\n\t}\n\n\t// 检查随机数生成\n\trandomGeneration, err := c.CheckRandomGeneration(filePath)\n\tif err != nil {\n\t\tresults[\"random_generation\"] = err.Error()\n\t} else {\n\t\tresults[\"random_generation\"] = randomGeneration\n\t}\n\n\t// 检查敏感数据\n\tsensitiveData, err := c.CheckSensitiveData(filePath)\n\tif err != nil {\n\t\tresults[\"sensitive_data\"] = err.Error()\n\t} else {\n\t\tresults[\"sensitive_data\"] = sensitiveData\n\t}\n\n\t// 检查沙箱逃逸\n\tsandboxEscape, err := c.CheckSandboxEscape(filePath)\n\tif err != nil {\n\t\tresults[\"sandbox_escape\"] = err.Error()\n\t} else {\n\t\tresults[\"sandbox_escape\"] = sandboxEscape\n\t}\n\n\treturn results, nil\n} " }, { "path": "go/internal/utils/security_test.go", "content": "package utils\n\nimport (\n\t\"os\"\n\t\"testing\"\n\t\"time\"\n)\n\nfunc TestNewSecurityChecker(t *testing.T) {\n\tchecker := NewSecurityChecker()\n\tif checker == nil {\n\t\tt.Error(\"NewSecurityChecker返回了nil\")\n\t}\n\n\tif len(checker.sensitivePatterns) == 0 {\n\t\tt.Error(\"敏感模式映射为空\")\n\t}\n\n\texpectedPatterns := []string{\"file_access\", \"network_access\", \"code_execution\", \"input_validation\", \"random_generation\", \"sensitive_data\"}\n\tfor _, pattern := range expectedPatterns {\n\t\tif patterns, ok := checker.sensitivePatterns[pattern]; !ok || len(patterns) == 0 {\n\t\t\tt.Errorf(\"缺少预期的模式类型: %s\", pattern)\n\t\t}\n\t}\n}\n\nfunc createTestFile(content string) (string, error) {\n\ttmpfile, err := os.CreateTemp(\"\", \"test_*.go\")\n\tif err != nil {\n\t\treturn \"\", err\n\t}\n\n\tif _, err := tmpfile.Write([]byte(content)); err != nil {\n\t\tos.Remove(tmpfile.Name())\n\t\treturn \"\", err\n\t}\n\n\tif err := tmpfile.Close(); err != nil {\n\t\tos.Remove(tmpfile.Name())\n\t\treturn \"\", err\n\t}\n\n\treturn tmpfile.Name(), nil\n}\n\nfunc TestCheckMemoryUsage(t *testing.T) {\n\tchecker := NewSecurityChecker()\n\tcontent := `package main\n\nimport \"fmt\"\n\nfunc main() {\n\tvar arr []int\n\tfor i := 0; i < 1000; i++ {\n\t\tarr = append(arr, i)\n\t}\n\tfmt.Println(arr)\n}`\n\n\tfilename, err := createTestFile(content)\n\tif err != nil {\n\t\tt.Fatalf(\"创建测试文件失败: %v\", err)\n\t}\n\tdefer os.Remove(filename)\n\n\tusage, err := checker.CheckMemoryUsage(filename)\n\tif err != nil {\n\t\tt.Errorf(\"检查内存使用失败: %v\", err)\n\t}\n\n\tif usage == 0 {\n\t\tt.Error(\"内存使用量不应为0\")\n\t}\n}\n\nfunc TestCheckExecutionTime(t *testing.T) {\n\tchecker := NewSecurityChecker()\n\tcontent := `package main\n\nimport \"time\"\n\nfunc main() {\n\ttime.Sleep(time.Second)\n}`\n\n\tfilename, err := createTestFile(content)\n\tif err != nil {\n\t\tt.Fatalf(\"创建测试文件失败: %v\", err)\n\t}\n\tdefer os.Remove(filename)\n\n\t// 测试正常超时\n\terr = checker.CheckExecutionTime(filename, 5*time.Second)\n\tif err != nil {\n\t\tt.Errorf(\"执行时间检查失败: %v\", err)\n\t}\n\n\t// 测试超时情况\n\terr = checker.CheckExecutionTime(filename, 1*time.Millisecond)\n\tif err == nil {\n\t\tt.Error(\"预期应该发生超时错误\")\n\t}\n}\n\nfunc TestCheckFileAccess(t *testing.T) {\n\tchecker := NewSecurityChecker()\n\tcontent := `package main\n\nimport (\n\t\"os\"\n\t\"io/ioutil\"\n)\n\nfunc main() {\n\tos.Open(\"test.txt\")\n\tioutil.ReadFile(\"config.json\")\n}`\n\n\tfilename, err := createTestFile(content)\n\tif err != nil {\n\t\tt.Fatalf(\"创建测试文件失败: %v\", err)\n\t}\n\tdefer os.Remove(filename)\n\n\tviolations, err := checker.CheckFileAccess(filename)\n\tif err != nil {\n\t\tt.Errorf(\"文件访问检查失败: %v\", err)\n\t}\n\n\tif len(violations) == 0 {\n\t\tt.Error(\"应该检测到文件访问违规\")\n\t}\n}\n\nfunc TestCheckNetworkAccess(t *testing.T) {\n\tchecker := NewSecurityChecker()\n\tcontent := `package main\n\nimport (\n\t\"net\"\n\t\"net/http\"\n)\n\nfunc main() {\n\tnet.Dial(\"tcp\", \"localhost:8080\")\n\thttp.Get(\"http://example.com\")\n}`\n\n\tfilename, err := createTestFile(content)\n\tif err != nil {\n\t\tt.Fatalf(\"创建测试文件失败: %v\", err)\n\t}\n\tdefer os.Remove(filename)\n\n\tviolations, err := checker.CheckNetworkAccess(filename)\n\tif err != nil {\n\t\tt.Errorf(\"网络访问检查失败: %v\", err)\n\t}\n\n\tif len(violations) == 0 {\n\t\tt.Error(\"应该检测到网络访问违规\")\n\t}\n}\n\nfunc TestCheckInputValidation(t *testing.T) {\n\tchecker := NewSecurityChecker()\n\tcontent := `package main\n\nimport (\n\t\"fmt\"\n\t\"bufio\"\n\t\"os\"\n)\n\nfunc main() {\n\tvar input string\n\tfmt.Scanln(&input)\n\tscanner := bufio.NewScanner(os.Stdin)\n}`\n\n\tfilename, err := createTestFile(content)\n\tif err != nil {\n\t\tt.Fatalf(\"创建测试文件失败: %v\", err)\n\t}\n\tdefer os.Remove(filename)\n\n\tissues, err := checker.CheckInputValidation(filename)\n\tif err != nil {\n\t\tt.Errorf(\"输入验证检查失败: %v\", err)\n\t}\n\n\tif len(issues) == 0 {\n\t\tt.Error(\"应该检测到未验证的输入\")\n\t}\n}\n\nfunc TestCheckRandomGeneration(t *testing.T) {\n\tchecker := NewSecurityChecker()\n\tcontent := `package main\n\nimport (\n\t\"math/rand\"\n\t\"crypto/rand\"\n)\n\nfunc main() {\n\trand.Int()\n\trand.Read(make([]byte, 32))\n}`\n\n\tfilename, err := createTestFile(content)\n\tif err != nil {\n\t\tt.Fatalf(\"创建测试文件失败: %v\", err)\n\t}\n\tdefer os.Remove(filename)\n\n\tissues, err := checker.CheckRandomGeneration(filename)\n\tif err != nil {\n\t\tt.Errorf(\"随机数生成检查失败: %v\", err)\n\t}\n\n\tif len(issues) == 0 {\n\t\tt.Error(\"应该检测到不安全的随机数生成\")\n\t}\n}\n\nfunc TestCheckSensitiveData(t *testing.T) {\n\tchecker := NewSecurityChecker()\n\tcontent := `package main\n\nimport \"fmt\"\n\nfunc main() {\n\tpassword := \"secret123\"\n\tfmt.Printf(\"Password: %s\\n\", password)\n}`\n\n\tfilename, err := createTestFile(content)\n\tif err != nil {\n\t\tt.Fatalf(\"创建测试文件失败: %v\", err)\n\t}\n\tdefer os.Remove(filename)\n\n\tissues, err := checker.CheckSensitiveData(filename)\n\tif err != nil {\n\t\tt.Errorf(\"敏感数据检查失败: %v\", err)\n\t}\n\n\tif len(issues) == 0 {\n\t\tt.Error(\"应该检测到敏感数据泄露风险\")\n\t}\n}\n\nfunc TestCheckSandboxEscape(t *testing.T) {\n\tchecker := NewSecurityChecker()\n\tcontent := `package main\n\nimport (\n\t\"os\"\n\t\"os/exec\"\n)\n\nfunc main() {\n\tos.Remove(\"test.txt\")\n\texec.Command(\"ls\").Run()\n}`\n\n\tfilename, err := createTestFile(content)\n\tif err != nil {\n\t\tt.Fatalf(\"创建测试文件失败: %v\", err)\n\t}\n\tdefer os.Remove(filename)\n\n\tviolations, err := checker.CheckSandboxEscape(filename)\n\tif err != nil {\n\t\tt.Errorf(\"沙箱逃逸检查失败: %v\", err)\n\t}\n\n\tif len(violations) == 0 {\n\t\tt.Error(\"应该检测到沙箱逃逸风险\")\n\t}\n}\n\nfunc TestPerformFullCheck(t *testing.T) {\n\tchecker := NewSecurityChecker()\n\tcontent := `package main\n\nimport (\n\t\"fmt\"\n\t\"os\"\n\t\"net/http\"\n\t\"math/rand\"\n)\n\nfunc main() {\n\tpassword := \"secret123\"\n\tos.Open(\"test.txt\")\n\thttp.Get(\"http://example.com\")\n\trand.Int()\n\tfmt.Printf(\"Password: %s\\n\", password)\n}`\n\n\tfilename, err := createTestFile(content)\n\tif err != nil {\n\t\tt.Fatalf(\"创建测试文件失败: %v\", err)\n\t}\n\tdefer os.Remove(filename)\n\n\tresults, err := checker.PerformFullCheck(filename)\n\tif err != nil {\n\t\tt.Errorf(\"完整检查失败: %v\", err)\n\t}\n\n\texpectedChecks := []string{\n\t\t\"memory_usage\",\n\t\t\"execution_time\",\n\t\t\"file_access\",\n\t\t\"network_access\",\n\t\t\"input_validation\",\n\t\t\"random_generation\",\n\t\t\"sensitive_data\",\n\t\t\"sandbox_escape\",\n\t}\n\n\tfor _, check := range expectedChecks {\n\t\tif _, ok := results[check]; !ok {\n\t\t\tt.Errorf(\"缺少检查结果: %s\", check)\n\t\t}\n\t}\n} " }, { "path": "go/internal/utils/tests/security_test.go", "content": "package utils\n\nimport (\n\t\"os\"\n\t\"path/filepath\"\n\t\"testing\"\n\t\"time\"\n\n\t\"github.com/stretchr/testify/assert\"\n)\n\nfunc TestSecurityChecker(t *testing.T) {\n\t// 创建临时目录\n\ttempDir, err := os.MkdirTemp(\"\", \"security_test\")\n\tif err != nil {\n\t\tt.Fatalf(\"创建临时目录失败: %v\", err)\n\t}\n\tdefer os.RemoveAll(tempDir)\n\n\t// 创建测试代码文件\n\ttestCode := `package main\n\nimport (\n\t\"crypto/rand\"\n\t\"database/sql\"\n\t\"fmt\"\n\t\"math/rand\"\n\t\"net/http\"\n\t\"os\"\n\t\"os/exec\"\n\t\"time\"\n)\n\nfunc unsafeMemory() {\n\t// 大量内存分配\n\tlargeSlice := make([]int, 1e7)\n\tfor i := range largeSlice {\n\t\tlargeSlice[i] = i\n\t}\n}\n\nfunc unsafeExecution() {\n\t// 长时间执行\n\ttime.Sleep(5 * time.Second)\n}\n\nfunc unsafeFileAccess() {\n\t// 危险的文件操作\n\tfile, _ := os.Open(\"/etc/passwd\")\n\tdefer file.Close()\n}\n\nfunc unsafeNetwork() {\n\t// 未验证的网络请求\n\thttp.Get(\"http://example.com\")\n}\n\nfunc unsafeInput() {\n\t// 未验证的输入\n\tvar input string\n\tfmt.Scanln(&input)\n\texec.Command(\"bash\", \"-c\", input).Run()\n}\n\nfunc unsafeRandom() {\n\t// 不安全的随机数生成\n\trand.Seed(time.Now().UnixNano())\n\tfmt.Println(rand.Int())\n}\n\nfunc unsafeSensitiveData() {\n\t// 敏感数据暴露\n\tpassword := \"super_secret_123\"\n\tfmt.Printf(\"Password: %s\\n\", password)\n}\n\nfunc unsafeSandbox() {\n\t// 沙箱逃逸尝试\n\texec.Command(\"rm\", \"-rf\", \"/\").Run()\n}\n`\n\n\ttestFile := filepath.Join(tempDir, \"test.go\")\n\terr = os.WriteFile(testFile, []byte(testCode), 0644)\n\tif err != nil {\n\t\tt.Fatalf(\"写入测试代码文件失败: %v\", err)\n\t}\n\n\t// 创建检查器实例\n\tchecker := NewSecurityChecker()\n\n\t// 测试内存使用检查\n\tt.Run(\"TestCheckMemoryUsage\", func(t *testing.T) {\n\t\tresult := checker.CheckMemoryUsage(testFile)\n\t\tassert.True(t, result.HasIssues)\n\t\tassert.Contains(t, result.Details, \"largeSlice\")\n\t\tassert.Greater(t, len(result.Patterns), 0)\n\t})\n\n\t// 测试执行时间检查\n\tt.Run(\"TestCheckExecutionTime\", func(t *testing.T) {\n\t\tresult := checker.CheckExecutionTime(testFile)\n\t\tassert.True(t, result.HasIssues)\n\t\tassert.Contains(t, result.Details, \"time.Sleep\")\n\t\tassert.Greater(t, len(result.Patterns), 0)\n\t})\n\n\t// 测试文件访问检查\n\tt.Run(\"TestCheckFileAccess\", func(t *testing.T) {\n\t\tresult := checker.CheckFileAccess(testFile)\n\t\tassert.True(t, result.HasIssues)\n\t\tassert.Contains(t, result.Details, \"/etc/passwd\")\n\t\tassert.Greater(t, len(result.Patterns), 0)\n\t})\n\n\t// 测试网络访问检查\n\tt.Run(\"TestCheckNetworkAccess\", func(t *testing.T) {\n\t\tresult := checker.CheckNetworkAccess(testFile)\n\t\tassert.True(t, result.HasIssues)\n\t\tassert.Contains(t, result.Details, \"http.Get\")\n\t\tassert.Greater(t, len(result.Patterns), 0)\n\t})\n\n\t// 测试输入验证检查\n\tt.Run(\"TestCheckInputValidation\", func(t *testing.T) {\n\t\tresult := checker.CheckInputValidation(testFile)\n\t\tassert.True(t, result.HasIssues)\n\t\tassert.Contains(t, result.Details, \"exec.Command\")\n\t\tassert.Greater(t, len(result.Patterns), 0)\n\t})\n\n\t// 测试随机数生成检查\n\tt.Run(\"TestCheckRandomGeneration\", func(t *testing.T) {\n\t\tresult := checker.CheckRandomGeneration(testFile)\n\t\tassert.True(t, result.HasIssues)\n\t\tassert.Contains(t, result.Details, \"math/rand\")\n\t\tassert.Greater(t, len(result.Patterns), 0)\n\t})\n\n\t// 测试敏感数据检查\n\tt.Run(\"TestCheckSensitiveData\", func(t *testing.T) {\n\t\tresult := checker.CheckSensitiveData(testFile)\n\t\tassert.True(t, result.HasIssues)\n\t\tassert.Contains(t, result.Details, \"password\")\n\t\tassert.Greater(t, len(result.Patterns), 0)\n\t})\n\n\t// 测试沙箱逃逸检查\n\tt.Run(\"TestCheckSandboxEscape\", func(t *testing.T) {\n\t\tresult := checker.CheckSandboxEscape(testFile)\n\t\tassert.True(t, result.HasIssues)\n\t\tassert.Contains(t, result.Details, \"exec.Command\")\n\t\tassert.Greater(t, len(result.Patterns), 0)\n\t})\n\n\t// 测试完整安全检查\n\tt.Run(\"TestPerformFullCheck\", func(t *testing.T) {\n\t\tresults := checker.PerformFullCheck(testFile)\n\t\tassert.NotNil(t, results)\n\t\tassert.Greater(t, len(results), 0)\n\n\t\t// 验证所有检查项都已执行\n\t\texpectedChecks := []string{\n\t\t\t\"memory_usage\",\n\t\t\t\"execution_time\",\n\t\t\t\"file_access\",\n\t\t\t\"network_access\",\n\t\t\t\"input_validation\",\n\t\t\t\"random_generation\",\n\t\t\t\"sensitive_data\",\n\t\t\t\"sandbox_escape\",\n\t\t}\n\n\t\tfor _, check := range expectedChecks {\n\t\t\tresult, ok := results[check]\n\t\t\tassert.True(t, ok)\n\t\t\tassert.True(t, result.HasIssues)\n\t\t\tassert.Greater(t, len(result.Patterns), 0)\n\t\t}\n\t})\n\n\t// 测试并发检查\n\tt.Run(\"TestConcurrentChecks\", func(t *testing.T) {\n\t\t// 创建多个测试文件\n\t\ttestFiles := make([]string, 5)\n\t\tfor i := range testFiles {\n\t\t\tfilePath := filepath.Join(tempDir, fmt.Sprintf(\"test_%d.go\", i))\n\t\t\terr := os.WriteFile(filePath, []byte(testCode), 0644)\n\t\t\tassert.NoError(t, err)\n\t\t\ttestFiles[i] = filePath\n\t\t}\n\n\t\t// 记录串行执行时间\n\t\tstartSerial := time.Now()\n\t\tfor _, file := range testFiles {\n\t\t\tchecker.PerformFullCheck(file)\n\t\t}\n\t\tserialDuration := time.Since(startSerial)\n\n\t\t// 记录并行执行时间\n\t\tstartParallel := time.Now()\n\t\tresultChan := make(chan map[string]SecurityCheckResult, len(testFiles))\n\t\tfor _, file := range testFiles {\n\t\t\tgo func(f string) {\n\t\t\t\tresultChan <- checker.PerformFullCheck(f)\n\t\t\t}(file)\n\t\t}\n\n\t\t// 收集结果\n\t\tresults := make([]map[string]SecurityCheckResult, 0, len(testFiles))\n\t\tfor i := 0; i < len(testFiles); i++ {\n\t\t\tresult := <-resultChan\n\t\t\tresults = append(results, result)\n\t\t}\n\t\tparallelDuration := time.Since(startParallel)\n\n\t\t// 验证结果\n\t\tassert.Equal(t, len(testFiles), len(results))\n\t\tfor _, result := range results {\n\t\t\tassert.NotNil(t, result)\n\t\t\tassert.Greater(t, len(result), 0)\n\t\t}\n\n\t\t// 验证并行执行更快\n\t\tassert.Less(t, parallelDuration, serialDuration)\n\t})\n\n\t// 测试错误处理\n\tt.Run(\"TestErrorHandling\", func(t *testing.T) {\n\t\t// 测试不存在的文件\n\t\tnonExistentFile := filepath.Join(tempDir, \"non_existent.go\")\n\t\tresult := checker.PerformFullCheck(nonExistentFile)\n\t\tassert.NotNil(t, result)\n\t\tfor _, check := range result {\n\t\t\tassert.False(t, check.HasIssues)\n\t\t\tassert.Contains(t, check.Details, \"file not found\")\n\t\t}\n\n\t\t// 测试无效的Go代码\n\t\tinvalidCode := \"invalid go code\"\n\t\tinvalidFile := filepath.Join(tempDir, \"invalid.go\")\n\t\terr := os.WriteFile(invalidFile, []byte(invalidCode), 0644)\n\t\tassert.NoError(t, err)\n\n\t\tresult = checker.PerformFullCheck(invalidFile)\n\t\tassert.NotNil(t, result)\n\t\tfor _, check := range result {\n\t\t\tassert.False(t, check.HasIssues)\n\t\t\tassert.Contains(t, check.Details, \"parse error\")\n\t\t}\n\t})\n}\n\nfunc TestSecurityCheckerEdgeCases(t *testing.T) {\n\tchecker := NewSecurityChecker()\n\n\t// 测试空文件\n\tt.Run(\"TestEmptyFile\", func(t *testing.T) {\n\t\ttempDir, err := os.MkdirTemp(\"\", \"empty_test\")\n\t\tassert.NoError(t, err)\n\t\tdefer os.RemoveAll(tempDir)\n\n\t\temptyFile := filepath.Join(tempDir, \"empty.go\")\n\t\terr = os.WriteFile(emptyFile, []byte(\"\"), 0644)\n\t\tassert.NoError(t, err)\n\n\t\tresult := checker.PerformFullCheck(emptyFile)\n\t\tassert.NotNil(t, result)\n\t\tfor _, check := range result {\n\t\t\tassert.False(t, check.HasIssues)\n\t\t}\n\t})\n\n\t// 测试大文件处理\n\tt.Run(\"TestLargeFile\", func(t *testing.T) {\n\t\ttempDir, err := os.MkdirTemp(\"\", \"large_test\")\n\t\tassert.NoError(t, err)\n\t\tdefer os.RemoveAll(tempDir)\n\n\t\t// 生成大文件\n\t\tlargeCode := `package main\n\nimport \"fmt\"\n\nfunc main() {\n`\n\t\tfor i := 0; i < 10000; i++ {\n\t\t\tlargeCode += fmt.Sprintf(\"\\tfmt.Println(%d)\\n\", i)\n\t\t}\n\t\tlargeCode += \"}\\n\"\n\n\t\tlargeFile := filepath.Join(tempDir, \"large.go\")\n\t\terr = os.WriteFile(largeFile, []byte(largeCode), 0644)\n\t\tassert.NoError(t, err)\n\n\t\tstartTime := time.Now()\n\t\tresult := checker.PerformFullCheck(largeFile)\n\t\tduration := time.Since(startTime)\n\n\t\tassert.NotNil(t, result)\n\t\tassert.Less(t, duration, 30*time.Second) // 确保大文件处理不会超时\n\t})\n\n\t// 测试并发限制\n\tt.Run(\"TestConcurrencyLimit\", func(t *testing.T) {\n\t\ttempDir, err := os.MkdirTemp(\"\", \"concurrency_test\")\n\t\tassert.NoError(t, err)\n\t\tdefer os.RemoveAll(tempDir)\n\n\t\t// 创建多个测试文件\n\t\tnumFiles := 100\n\t\ttestFiles := make([]string, numFiles)\n\t\ttestCode := `package main\nfunc main() {}`\n\n\t\tfor i := range testFiles {\n\t\t\tfilePath := filepath.Join(tempDir, fmt.Sprintf(\"test_%d.go\", i))\n\t\t\terr := os.WriteFile(filePath, []byte(testCode), 0644)\n\t\t\tassert.NoError(t, err)\n\t\t\ttestFiles[i] = filePath\n\t\t}\n\n\t\t// 并发执行检查\n\t\tstartTime := time.Now()\n\t\tresultChan := make(chan map[string]SecurityCheckResult, numFiles)\n\t\tfor _, file := range testFiles {\n\t\t\tgo func(f string) {\n\t\t\t\tresultChan <- checker.PerformFullCheck(f)\n\t\t\t}(file)\n\t\t}\n\n\t\t// 收集结果\n\t\tresults := make([]map[string]SecurityCheckResult, 0, numFiles)\n\t\tfor i := 0; i < numFiles; i++ {\n\t\t\tresult := <-resultChan\n\t\t\tresults = append(results, result)\n\t\t}\n\t\tduration := time.Since(startTime)\n\n\t\tassert.Equal(t, numFiles, len(results))\n\t\tassert.Less(t, duration, 60*time.Second) // 确保并发处理不会超时\n\t})\n} " }, { "path": "go/internal/web/app.go", "content": "package web\n\nimport (\n\t\"fmt\"\n\t\"html/template\"\n\t\"io/ioutil\"\n\t\"net/http\"\n\t\"os\"\n\t\"path/filepath\"\n\t\"time\"\n\n\t\"github.com/gin-gonic/gin\"\n\t\"github.com/re-movery/re-movery/internal/core\"\n\t\"github.com/re-movery/re-movery/internal/detectors\"\n)\n\n// App is the web application\ntype App struct {\n\tscanner *core.Scanner\n\trouter *gin.Engine\n}\n\n// NewApp creates a new web application\nfunc NewApp() *App {\n\tapp := &App{\n\t\tscanner: core.NewScanner(),\n\t\trouter: gin.Default(),\n\t}\n\n\t// Register detectors\n\tapp.scanner.RegisterDetector(detectors.NewPythonDetector())\n\tapp.scanner.RegisterDetector(detectors.NewJavaScriptDetector())\n\n\t// Setup routes\n\tapp.setupRoutes()\n\n\treturn app\n}\n\n// setupRoutes sets up the routes for the web application\nfunc (a *App) setupRoutes() {\n\t// Serve static files\n\ta.router.Static(\"/static\", \"./static\")\n\n\t// Load templates\n\ta.router.LoadHTMLGlob(\"templates/*\")\n\n\t// Routes\n\ta.router.GET(\"/\", a.indexHandler)\n\ta.router.POST(\"/scan/file\", a.scanFileHandler)\n\ta.router.POST(\"/scan/directory\", a.scanDirectoryHandler)\n\ta.router.GET(\"/api/languages\", a.languagesHandler)\n\ta.router.GET(\"/health\", a.healthHandler)\n}\n\n// Run runs the web application\nfunc (a *App) Run(host string, port int) error {\n\treturn a.router.Run(fmt.Sprintf(\"%s:%d\", host, port))\n}\n\n// indexHandler handles the index page\nfunc (a *App) indexHandler(c *gin.Context) {\n\tc.HTML(http.StatusOK, \"index.html\", gin.H{\n\t\t\"title\": \"Re-movery - Security Scanner\",\n\t})\n}\n\n// scanFileHandler handles file scanning\nfunc (a *App) scanFileHandler(c *gin.Context) {\n\t// Get file from form\n\tfile, err := c.FormFile(\"file\")\n\tif err != nil {\n\t\tc.JSON(http.StatusBadRequest, gin.H{\n\t\t\t\"error\": \"No file provided\",\n\t\t})\n\t\treturn\n\t}\n\n\t// Save file to temporary location\n\ttempFile := filepath.Join(os.TempDir(), file.Filename)\n\tif err := c.SaveUploadedFile(file, tempFile); err != nil {\n\t\tc.JSON(http.StatusInternalServerError, gin.H{\n\t\t\t\"error\": \"Failed to save file\",\n\t\t})\n\t\treturn\n\t}\n\tdefer os.Remove(tempFile)\n\n\t// Scan file\n\tresults, err := a.scanner.ScanFile(tempFile)\n\tif err != nil {\n\t\tc.JSON(http.StatusInternalServerError, gin.H{\n\t\t\t\"error\": fmt.Sprintf(\"Failed to scan file: %v\", err),\n\t\t})\n\t\treturn\n\t}\n\n\t// Generate summary\n\tsummary := core.GenerateSummary(map[string][]core.Match{\n\t\tfile.Filename: results,\n\t})\n\n\t// Return results\n\tc.JSON(http.StatusOK, gin.H{\n\t\t\"results\": map[string][]core.Match{\n\t\t\tfile.Filename: results,\n\t\t},\n\t\t\"summary\": summary,\n\t})\n}\n\n// scanDirectoryHandler handles directory scanning\nfunc (a *App) scanDirectoryHandler(c *gin.Context) {\n\t// Get directory path from form\n\tdirectory := c.PostForm(\"directory\")\n\tif directory == \"\" {\n\t\tc.JSON(http.StatusBadRequest, gin.H{\n\t\t\t\"error\": \"No directory provided\",\n\t\t})\n\t\treturn\n\t}\n\n\t// Check if directory exists\n\tif _, err := os.Stat(directory); os.IsNotExist(err) {\n\t\tc.JSON(http.StatusBadRequest, gin.H{\n\t\t\t\"error\": \"Directory does not exist\",\n\t\t})\n\t\treturn\n\t}\n\n\t// Get exclude patterns\n\texcludePatterns := c.PostFormArray(\"exclude\")\n\n\t// Scan directory\n\tresults, err := a.scanner.ScanDirectory(directory, excludePatterns)\n\tif err != nil {\n\t\tc.JSON(http.StatusInternalServerError, gin.H{\n\t\t\t\"error\": fmt.Sprintf(\"Failed to scan directory: %v\", err),\n\t\t})\n\t\treturn\n\t}\n\n\t// Generate summary\n\tsummary := core.GenerateSummary(results)\n\n\t// Return results\n\tc.JSON(http.StatusOK, gin.H{\n\t\t\"results\": results,\n\t\t\"summary\": summary,\n\t})\n}\n\n// languagesHandler handles the supported languages request\nfunc (a *App) languagesHandler(c *gin.Context) {\n\tlanguages := a.scanner.SupportedLanguages()\n\tc.JSON(http.StatusOK, gin.H{\n\t\t\"languages\": languages,\n\t})\n}\n\n// healthHandler handles the health check request\nfunc (a *App) healthHandler(c *gin.Context) {\n\tc.JSON(http.StatusOK, gin.H{\n\t\t\"status\": \"ok\",\n\t\t\"time\": time.Now().Format(time.RFC3339),\n\t})\n} " }, { "path": "go/internal/web/static/css/style.css", "content": "/* Re-movery 样式文件 */\n\nbody {\n font-family: 'Segoe UI', Tahoma, Geneva, Verdana, sans-serif;\n background-color: #f8f9fa;\n}\n\n.navbar-brand {\n font-weight: bold;\n color: #0d6efd;\n}\n\n.card {\n border-radius: 10px;\n box-shadow: 0 4px 6px rgba(0, 0, 0, 0.1);\n margin-bottom: 20px;\n}\n\n.card-header {\n font-weight: bold;\n background-color: #f8f9fa;\n}\n\n.severity-high {\n color: #dc3545;\n}\n\n.severity-medium {\n color: #fd7e14;\n}\n\n.severity-low {\n color: #0dcaf0;\n}\n\n.chart-container {\n height: 300px;\n}\n\n.nav-pills .nav-link.active {\n background-color: #0d6efd;\n}\n\n.nav-pills .nav-link {\n color: #495057;\n}\n\n.file-item {\n cursor: pointer;\n}\n\n.file-item:hover {\n background-color: #f8f9fa;\n}\n\n.code-block {\n background-color: #f8f9fa;\n border-radius: 5px;\n padding: 10px;\n font-family: monospace;\n white-space: pre-wrap;\n margin-top: 10px;\n}\n\n.footer {\n margin-top: 50px;\n padding: 20px 0;\n background-color: #f8f9fa;\n text-align: center;\n color: #6c757d;\n}\n\n/* 按钮样式 */\n.btn-primary {\n background-color: #0d6efd;\n border-color: #0d6efd;\n}\n\n.btn-primary:hover {\n background-color: #0b5ed7;\n border-color: #0a58ca;\n}\n\n/* 表单样式 */\n.form-control:focus {\n border-color: #0d6efd;\n box-shadow: 0 0 0 0.25rem rgba(13, 110, 253, 0.25);\n}\n\n/* 表格样式 */\n.table {\n border-collapse: collapse;\n width: 100%;\n}\n\n.table th {\n background-color: #f8f9fa;\n font-weight: bold;\n}\n\n.table-striped tbody tr:nth-of-type(odd) {\n background-color: rgba(0, 0, 0, 0.05);\n}\n\n/* 徽章样式 */\n.badge {\n font-weight: normal;\n padding: 0.35em 0.65em;\n}\n\n/* 响应式调整 */\n@media (max-width: 768px) {\n .chart-container {\n height: 200px;\n }\n} " }, { "path": "go/internal/web/static/js/app.js", "content": "/**\n * Re-movery Web应用JavaScript\n */\n\n// 全局变量\nlet severityChart = null;\nlet vulnerabilityChart = null;\n\n// 初始化应用\ndocument.addEventListener('DOMContentLoaded', function() {\n // 初始化图表\n initCharts();\n \n // 加载设置\n loadSettings();\n \n // 绑定事件\n bindEvents();\n});\n\n// 初始化图表\nfunction initCharts() {\n // 漏洞严重程度分布图\n const severityCtx = document.getElementById('severity-chart').getContext('2d');\n severityChart = new Chart(severityCtx, {\n type: 'pie',\n data: {\n labels: ['高危', '中危', '低危'],\n datasets: [{\n data: [0, 0, 0],\n backgroundColor: ['#dc3545', '#fd7e14', '#0dcaf0']\n }]\n },\n options: {\n responsive: true,\n plugins: {\n legend: {\n position: 'bottom'\n }\n }\n }\n });\n\n // 常见漏洞类型图\n const vulnerabilityCtx = document.getElementById('vulnerability-chart').getContext('2d');\n vulnerabilityChart = new Chart(vulnerabilityCtx, {\n type: 'bar',\n data: {\n labels: [],\n datasets: [{\n label: '漏洞数量',\n data: [],\n backgroundColor: '#0d6efd'\n }]\n },\n options: {\n responsive: true,\n plugins: {\n legend: {\n display: false\n }\n },\n scales: {\n y: {\n beginAtZero: true,\n ticks: {\n precision: 0\n }\n }\n }\n }\n });\n}\n\n// 加载设置\nfunction loadSettings() {\n const settings = JSON.parse(localStorage.getItem('re-movery-settings') || '{}');\n document.getElementById('confidence-threshold').value = settings.confidenceThreshold || 0.7;\n document.getElementById('confidence-value').textContent = settings.confidenceThreshold || 0.7;\n document.getElementById('default-parallel').checked = settings.defaultParallel !== false;\n document.getElementById('default-incremental').checked = settings.defaultIncremental !== false;\n \n // 应用设置到扫描表单\n document.getElementById('parallel').checked = settings.defaultParallel !== false;\n document.getElementById('incremental').checked = settings.defaultIncremental !== false;\n}\n\n// 保存设置\nfunction saveSettings() {\n const settings = {\n confidenceThreshold: parseFloat(document.getElementById('confidence-threshold').value),\n defaultParallel: document.getElementById('default-parallel').checked,\n defaultIncremental: document.getElementById('default-incremental').checked\n };\n localStorage.setItem('re-movery-settings', JSON.stringify(settings));\n alert('设置已保存');\n \n // 应用设置到扫描表单\n document.getElementById('parallel').checked = settings.defaultParallel;\n document.getElementById('incremental').checked = settings.defaultIncremental;\n}\n\n// 绑定事件\nfunction bindEvents() {\n // 设置表单提交\n document.getElementById('settings-form').addEventListener('submit', function(e) {\n e.preventDefault();\n saveSettings();\n });\n\n // 更新置信度值显示\n document.getElementById('confidence-threshold').addEventListener('input', function() {\n document.getElementById('confidence-value').textContent = this.value;\n });\n\n // 文件扫描表单提交\n document.getElementById('file-scan-form').addEventListener('submit', function(e) {\n e.preventDefault();\n scanFile();\n });\n\n // 目录扫描表单提交\n document.getElementById('directory-scan-form').addEventListener('submit', function(e) {\n e.preventDefault();\n scanDirectory();\n });\n}\n\n// 扫描文件\nfunction scanFile() {\n const fileInput = document.getElementById('file');\n if (!fileInput.files.length) {\n alert('请选择文件');\n return;\n }\n\n const formData = new FormData();\n formData.append('file', fileInput.files[0]);\n\n // 显示加载指示器\n showLoading('正在扫描文件...');\n\n fetch('/scan/file', {\n method: 'POST',\n body: formData\n })\n .then(response => {\n if (!response.ok) {\n throw new Error('扫描请求失败');\n }\n return response.json();\n })\n .then(data => {\n hideLoading();\n updateResults(data);\n // 切换到结果标签页\n document.querySelector('a[href=\"#results\"]').click();\n })\n .catch(error => {\n hideLoading();\n console.error('Error:', error);\n alert('扫描失败: ' + error.message);\n });\n}\n\n// 扫描目录\nfunction scanDirectory() {\n const directory = document.getElementById('directory').value;\n if (!directory) {\n alert('请输入目录路径');\n return;\n }\n\n const formData = new FormData();\n formData.append('directory', directory);\n \n const exclude = document.getElementById('exclude').value;\n if (exclude) {\n exclude.split(',').forEach(pattern => {\n formData.append('exclude', pattern.trim());\n });\n }\n \n formData.append('parallel', document.getElementById('parallel').checked);\n formData.append('incremental', document.getElementById('incremental').checked);\n\n // 显示加载指示器\n showLoading('正在扫描目录...');\n\n fetch('/scan/directory', {\n method: 'POST',\n body: formData\n })\n .then(response => {\n if (!response.ok) {\n throw new Error('扫描请求失败');\n }\n return response.json();\n })\n .then(data => {\n hideLoading();\n updateResults(data);\n // 切换到结果标签页\n document.querySelector('a[href=\"#results\"]').click();\n })\n .catch(error => {\n hideLoading();\n console.error('Error:', error);\n alert('扫描失败: ' + error.message);\n });\n}\n\n// 更新结果\nfunction updateResults(data) {\n // 更新计数\n document.getElementById('high-count').textContent = data.summary.high;\n document.getElementById('medium-count').textContent = data.summary.medium;\n document.getElementById('low-count').textContent = data.summary.low;\n\n // 更新图表\n severityChart.data.datasets[0].data = [\n data.summary.high,\n data.summary.medium,\n data.summary.low\n ];\n severityChart.update();\n\n // 更新漏洞类型图表\n const vulnerabilities = data.summary.vulnerabilities || {};\n const sortedVulns = Object.entries(vulnerabilities)\n .sort((a, b) => b[1] - a[1])\n .slice(0, 10);\n \n vulnerabilityChart.data.labels = sortedVulns.map(v => v[0]);\n vulnerabilityChart.data.datasets[0].data = sortedVulns.map(v => v[1]);\n vulnerabilityChart.update();\n\n // 更新结果列表\n const resultsContainer = document.getElementById('results-container');\n resultsContainer.innerHTML = '';\n\n if (Object.keys(data.results).length === 0) {\n resultsContainer.innerHTML = `\n
\n 未发现漏洞。\n
\n `;\n return;\n }\n\n for (const [filePath, matches] of Object.entries(data.results)) {\n if (matches.length === 0) continue;\n\n const fileCard = document.createElement('div');\n fileCard.className = 'card mb-3';\n \n const fileHeader = document.createElement('div');\n fileHeader.className = 'card-header file-item';\n fileHeader.innerHTML = `\n ${filePath}\n ${matches.length}\n `;\n \n const fileContent = document.createElement('div');\n fileContent.className = 'card-body';\n fileContent.style.display = 'none';\n \n // 添加漏洞列表\n const table = document.createElement('table');\n table.className = 'table table-striped';\n table.innerHTML = `\n \n \n 行号\n 严重程度\n 漏洞\n 置信度\n \n \n \n ${matches.map(match => `\n \n ${match.lineNumber}\n \n \n ${getSeverityText(match.signature.severity)}\n \n \n \n ${match.signature.name}\n

${match.signature.description}

\n
${escapeHtml(match.matchedCode)}
\n \n ${Math.round(match.confidence * 100)}%\n \n `).join('')}\n \n `;\n \n fileContent.appendChild(table);\n fileCard.appendChild(fileHeader);\n fileCard.appendChild(fileContent);\n resultsContainer.appendChild(fileCard);\n \n // 添加点击事件\n fileHeader.addEventListener('click', function() {\n if (fileContent.style.display === 'none') {\n fileContent.style.display = 'block';\n } else {\n fileContent.style.display = 'none';\n }\n });\n }\n}\n\n// 获取严重程度样式类\nfunction getSeverityClass(severity) {\n switch (severity.toLowerCase()) {\n case 'high': return 'bg-danger';\n case 'medium': return 'bg-warning text-dark';\n case 'low': return 'bg-info text-dark';\n default: return 'bg-secondary';\n }\n}\n\n// 获取严重程度文本\nfunction getSeverityText(severity) {\n switch (severity.toLowerCase()) {\n case 'high': return '高危';\n case 'medium': return '中危';\n case 'low': return '低危';\n default: return severity;\n }\n}\n\n// HTML转义\nfunction escapeHtml(html) {\n const div = document.createElement('div');\n div.textContent = html;\n return div.innerHTML;\n}\n\n// 显示加载指示器\nfunction showLoading(message) {\n let loadingDiv = document.getElementById('loading-indicator');\n if (!loadingDiv) {\n loadingDiv = document.createElement('div');\n loadingDiv.id = 'loading-indicator';\n loadingDiv.className = 'position-fixed top-0 start-0 w-100 h-100 d-flex justify-content-center align-items-center bg-white bg-opacity-75';\n loadingDiv.style.zIndex = '9999';\n loadingDiv.innerHTML = `\n
\n \n

${message || '加载中...'}

\n
\n `;\n document.body.appendChild(loadingDiv);\n } else {\n document.getElementById('loading-message').textContent = message || '加载中...';\n loadingDiv.style.display = 'flex';\n }\n}\n\n// 隐藏加载指示器\nfunction hideLoading() {\n const loadingDiv = document.getElementById('loading-indicator');\n if (loadingDiv) {\n loadingDiv.style.display = 'none';\n }\n} " }, { "path": "go/internal/web/templates/index.html", "content": "\n\n\n \n \n Re-movery - 安全漏洞扫描工具\n \n \n \n\n\n \n \n\n \n
\n
\n \n
\n

仪表盘

\n
\n
\n
\n
\n 高危漏洞\n
\n
\n

0

\n
\n
\n
\n
\n
\n
\n 中危漏洞\n
\n
\n

0

\n
\n
\n
\n
\n
\n
\n 低危漏洞\n
\n
\n

0

\n
\n
\n
\n
\n
\n
\n
\n
\n 漏洞严重程度分布\n
\n
\n
\n \n
\n
\n
\n
\n
\n
\n
\n 常见漏洞类型\n
\n
\n
\n \n
\n
\n
\n
\n
\n
\n\n \n
\n

扫描

\n
\n
\n
\n
\n 扫描文件\n
\n
\n
\n
\n \n \n
\n \n
\n
\n
\n
\n
\n
\n
\n 扫描目录\n
\n
\n
\n
\n \n \n
\n
\n \n \n
\n
\n \n \n
\n
\n \n \n
\n \n
\n
\n
\n
\n
\n
\n\n \n
\n

扫描结果

\n
\n
\n 漏洞列表\n
\n
\n
\n
\n 请先扫描文件或目录以查看结果。\n
\n
\n
\n
\n
\n\n \n
\n

设置

\n
\n
\n 扫描设置\n
\n
\n
\n
\n \n \n
0.7
\n
\n
\n \n \n
\n
\n \n \n
\n \n
\n
\n
\n
\n
\n
\n\n \n
\n
\n

Re-movery - 安全漏洞扫描工具 © 2023

\n
\n
\n\n \n \n \n \n\n " }, { "path": "go/tests/integration/workflow_test.go", "content": "package integration\n\nimport (\n\t\"encoding/json\"\n\t\"os\"\n\t\"path/filepath\"\n\t\"testing\"\n\t\"time\"\n\n\t\"github.com/stretchr/testify/assert\"\n\n\t\"github.com/heyangxu/Re-movery/go/internal/analyzers\"\n\t\"github.com/heyangxu/Re-movery/go/internal/detectors\"\n\t\"github.com/heyangxu/Re-movery/go/internal/reporters\"\n\t\"github.com/heyangxu/Re-movery/go/internal/utils\"\n)\n\nfunc TestWorkflow(t *testing.T) {\n\t// 创建临时目录\n\ttempDir, err := os.MkdirTemp(\"\", \"workflow_test\")\n\tif err != nil {\n\t\tt.Fatalf(\"创建临时目录失败: %v\", err)\n\t}\n\tdefer os.RemoveAll(tempDir)\n\n\t// 创建测试项目结构\n\terr = createTestProject(tempDir)\n\tif err != nil {\n\t\tt.Fatalf(\"创建测试项目失败: %v\", err)\n\t}\n\n\t// 初始化组件\n\tdetector := detectors.NewVulnerabilityDetector()\n\tchecker := utils.NewSecurityChecker()\n\tanalyzer := analyzers.NewCodeAnalyzer()\n\treporter := reporters.NewHTMLReporter()\n\n\t// 测试完整工作流程\n\tt.Run(\"TestFullWorkflow\", func(t *testing.T) {\n\t\t// 加载配置\n\t\tconfigFile := filepath.Join(tempDir, \"config.json\")\n\t\tconfigData, err := os.ReadFile(configFile)\n\t\tassert.NoError(t, err)\n\n\t\tvar config map[string]interface{}\n\t\terr = json.Unmarshal(configData, &config)\n\t\tassert.NoError(t, err)\n\n\t\t// 加载签名\n\t\tsignatureFile := filepath.Join(tempDir, \"signatures.json\")\n\t\terr = detector.LoadSignatures(signatureFile)\n\t\tassert.NoError(t, err)\n\n\t\t// 分析源代码文件\n\t\tsrcDir := filepath.Join(tempDir, \"src\")\n\t\tvulnerableFile := filepath.Join(srcDir, \"vulnerable.go\")\n\t\tsafeFile := filepath.Join(srcDir, \"safe.go\")\n\n\t\t// 检测漏洞\n\t\tvulnerableMatches, err := detector.DetectFile(vulnerableFile)\n\t\tassert.NoError(t, err)\n\t\tsafeMatches, err := detector.DetectFile(safeFile)\n\t\tassert.NoError(t, err)\n\n\t\tassert.Greater(t, len(vulnerableMatches), 0)\n\t\tassert.Equal(t, 0, len(safeMatches))\n\n\t\t// 执行安全检查\n\t\tvulnerableSecurity := checker.PerformFullCheck(vulnerableFile)\n\t\tsafeSecurity := checker.PerformFullCheck(safeFile)\n\n\t\tassert.True(t, hasIssues(vulnerableSecurity))\n\t\tassert.False(t, hasIssues(safeSecurity))\n\n\t\t// 代码分析\n\t\tvulnerableAnalysis, err := analyzer.AnalyzeFile(vulnerableFile)\n\t\tassert.NoError(t, err)\n\t\tsafeAnalysis, err := analyzer.AnalyzeFile(safeFile)\n\t\tassert.NoError(t, err)\n\n\t\tassert.Greater(t, vulnerableAnalysis.Complexity, safeAnalysis.Complexity)\n\n\t\t// 生成报告\n\t\treportData := map[string]interface{}{\n\t\t\t\"project_name\": config[\"project_name\"],\n\t\t\t\"scan_time\": time.Now().Format(\"2006-01-02 15:04:05\"),\n\t\t\t\"files_scanned\": []string{vulnerableFile, safeFile},\n\t\t\t\"vulnerability_results\": map[string]interface{}{\n\t\t\t\t\"vulnerable.go\": vulnerableMatches,\n\t\t\t\t\"safe.go\": safeMatches,\n\t\t\t},\n\t\t\t\"security_results\": map[string]interface{}{\n\t\t\t\t\"vulnerable.go\": vulnerableSecurity,\n\t\t\t\t\"safe.go\": safeSecurity,\n\t\t\t},\n\t\t\t\"analysis_results\": map[string]interface{}{\n\t\t\t\t\"vulnerable.go\": vulnerableAnalysis,\n\t\t\t\t\"safe.go\": safeAnalysis,\n\t\t\t},\n\t\t}\n\n\t\treportFile := filepath.Join(tempDir, \"reports\", \"report.html\")\n\t\terr = reporter.GenerateReport(reportData, reportFile)\n\t\tassert.NoError(t, err)\n\n\t\tassert.FileExists(t, reportFile)\n\t\tfileInfo, err := os.Stat(reportFile)\n\t\tassert.NoError(t, err)\n\t\tassert.Greater(t, fileInfo.Size(), int64(0))\n\t})\n\n\t// 测试并行处理\n\tt.Run(\"TestParallelProcessing\", func(t *testing.T) {\n\t\t// 创建多个测试文件\n\t\tsrcDir := filepath.Join(tempDir, \"src\")\n\t\ttestFiles := make([]string, 5)\n\t\ttestCode := `package main\n\nimport \"os/exec\"\n\nfunc main() {\n\texec.Command(\"ls\").Run()\n}\n`\n\t\tfor i := range testFiles {\n\t\t\tfilePath := filepath.Join(srcDir, \"test_%d.go\")\n\t\t\terr := os.WriteFile(filePath, []byte(testCode), 0644)\n\t\t\tassert.NoError(t, err)\n\t\t\ttestFiles[i] = filePath\n\t\t}\n\n\t\t// 串行处理时间\n\t\tstartSerial := time.Now()\n\t\tfor _, file := range testFiles {\n\t\t\t_, err := detector.DetectFile(file)\n\t\t\tassert.NoError(t, err)\n\t\t\tchecker.PerformFullCheck(file)\n\t\t\t_, err = analyzer.AnalyzeFile(file)\n\t\t\tassert.NoError(t, err)\n\t\t}\n\t\tserialDuration := time.Since(startSerial)\n\n\t\t// 并行处理时间\n\t\tstartParallel := time.Now()\n\t\tresultChan := make(chan struct{}, len(testFiles))\n\t\tfor _, file := range testFiles {\n\t\t\tgo func(f string) {\n\t\t\t\t_, err := detector.DetectFile(f)\n\t\t\t\tassert.NoError(t, err)\n\t\t\t\tchecker.PerformFullCheck(f)\n\t\t\t\t_, err = analyzer.AnalyzeFile(f)\n\t\t\t\tassert.NoError(t, err)\n\t\t\t\tresultChan <- struct{}{}\n\t\t\t}(file)\n\t\t}\n\n\t\t// 等待所有并行任务完成\n\t\tfor i := 0; i < len(testFiles); i++ {\n\t\t\t<-resultChan\n\t\t}\n\t\tparallelDuration := time.Since(startParallel)\n\n\t\tassert.Less(t, parallelDuration, serialDuration)\n\t})\n\n\t// 测试错误处理\n\tt.Run(\"TestErrorHandling\", func(t *testing.T) {\n\t\t// 测试无效的配置文件\n\t\tinvalidConfig := filepath.Join(tempDir, \"invalid_config.json\")\n\t\terr := os.WriteFile(invalidConfig, []byte(\"invalid json\"), 0644)\n\t\tassert.NoError(t, err)\n\n\t\t_, err = os.ReadFile(invalidConfig)\n\t\tassert.NoError(t, err)\n\t\tvar config map[string]interface{}\n\t\terr = json.Unmarshal([]byte(\"invalid json\"), &config)\n\t\tassert.Error(t, err)\n\n\t\t// 测试不存在的源代码文件\n\t\tnonExistentFile := filepath.Join(tempDir, \"non_existent.go\")\n\t\t_, err = detector.DetectFile(nonExistentFile)\n\t\tassert.Error(t, err)\n\n\t\t// 测试无效的源代码\n\t\tinvalidCode := filepath.Join(tempDir, \"invalid.go\")\n\t\terr = os.WriteFile(invalidCode, []byte(\"invalid go code\"), 0644)\n\t\tassert.NoError(t, err)\n\n\t\t_, err = analyzer.AnalyzeFile(invalidCode)\n\t\tassert.Error(t, err)\n\t})\n}\n\nfunc createTestProject(dir string) error {\n\t// 创建配置文件\n\tconfig := map[string]interface{}{\n\t\t\"project_name\": \"Test Project\",\n\t\t\"scan_paths\": []string{\"src\"},\n\t\t\"exclude_paths\": []string{\"tests\", \"docs\"},\n\t\t\"report_format\": \"html\",\n\t\t\"report_path\": \"reports\",\n\t\t\"severity_threshold\": \"medium\",\n\t\t\"parallel_processing\": true,\n\t\t\"max_workers\": 4,\n\t}\n\n\tconfigFile := filepath.Join(dir, \"config.json\")\n\tconfigData, err := json.MarshalIndent(config, \"\", \" \")\n\tif err != nil {\n\t\treturn err\n\t}\n\n\terr = os.WriteFile(configFile, configData, 0644)\n\tif err != nil {\n\t\treturn err\n\t}\n\n\t// 创建签名文件\n\tsignatures := map[string]interface{}{\n\t\t\"signatures\": []map[string]interface{}{\n\t\t\t{\n\t\t\t\t\"id\": \"CMD001\",\n\t\t\t\t\"name\": \"命令注入\",\n\t\t\t\t\"severity\": \"high\",\n\t\t\t\t\"code_patterns\": []string{\n\t\t\t\t\t`exec\\.Command\\([^)]*\\)`,\n\t\t\t\t\t`os\\.exec\\.Command\\([^)]*\\)`,\n\t\t\t\t},\n\t\t\t},\n\t\t\t{\n\t\t\t\t\"id\": \"SQL001\",\n\t\t\t\t\"name\": \"SQL注入\",\n\t\t\t\t\"severity\": \"high\",\n\t\t\t\t\"code_patterns\": []string{\n\t\t\t\t\t`db\\.Query\\([^)]*\\+[^)]*\\)`,\n\t\t\t\t\t`db\\.Exec\\([^)]*\\+[^)]*\\)`,\n\t\t\t\t},\n\t\t\t},\n\t\t},\n\t}\n\n\tsignatureFile := filepath.Join(dir, \"signatures.json\")\n\tsignatureData, err := json.MarshalIndent(signatures, \"\", \" \")\n\tif err != nil {\n\t\treturn err\n\t}\n\n\terr = os.WriteFile(signatureFile, signatureData, 0644)\n\tif err != nil {\n\t\treturn err\n\t}\n\n\t// 创建源代码目录\n\tsrcDir := filepath.Join(dir, \"src\")\n\terr = os.MkdirAll(srcDir, 0755)\n\tif err != nil {\n\t\treturn err\n\t}\n\n\t// 创建漏洞代码文件\n\tvulnerableCode := `package main\n\nimport (\n\t\"database/sql\"\n\t\"os/exec\"\n)\n\nfunc unsafeCommand(cmd string) {\n\texec.Command(\"bash\", \"-c\", cmd).Run()\n}\n\nfunc unsafeQuery(db *sql.DB, id string) {\n\tdb.Query(\"SELECT * FROM users WHERE id = \" + id)\n}\n\nfunc main() {\n\tunsafeCommand(\"ls -l\")\n\tdb, _ := sql.Open(\"mysql\", \"user:password@/dbname\")\n\tunsafeQuery(db, \"1 OR 1=1\")\n}\n`\n\n\tvulnerableFile := filepath.Join(srcDir, \"vulnerable.go\")\n\terr = os.WriteFile(vulnerableFile, []byte(vulnerableCode), 0644)\n\tif err != nil {\n\t\treturn err\n\t}\n\n\t// 创建安全代码文件\n\tsafeCode := `package main\n\nimport (\n\t\"database/sql\"\n)\n\nfunc safeQuery(db *sql.DB, id string) {\n\tdb.Query(\"SELECT * FROM users WHERE id = ?\", id)\n}\n\nfunc main() {\n\tdb, _ := sql.Open(\"mysql\", \"user:password@/dbname\")\n\tsafeQuery(db, \"1\")\n}\n`\n\n\tsafeFile := filepath.Join(srcDir, \"safe.go\")\n\terr = os.WriteFile(safeFile, []byte(safeCode), 0644)\n\tif err != nil {\n\t\treturn err\n\t}\n\n\t// 创建报告目录\n\treportDir := filepath.Join(dir, \"reports\")\n\treturn os.MkdirAll(reportDir, 0755)\n}\n\nfunc hasIssues(results map[string]utils.SecurityCheckResult) bool {\n\tfor _, result := range results {\n\t\tif result.HasIssues {\n\t\t\treturn true\n\t\t}\n\t}\n\treturn false\n} " }, { "path": "go/tests/security/security_test.go", "content": "package security\n\nimport (\n\t\"fmt\"\n\t\"io/ioutil\"\n\t\"os\"\n\t\"path/filepath\"\n\t\"testing\"\n\t\"time\"\n\n\t\"github.com/stretchr/testify/assert\"\n\t\"github.com/stretchr/testify/require\"\n\n\t\"re-movery/internal/detectors\"\n\t\"re-movery/internal/utils\"\n)\n\n// TestSecurity 包含所有安全相关的测试\ntype TestSecurity struct {\n\ttempDir string\n\tdetector *detectors.VulnerabilityDetector\n\tchecker *utils.SecurityChecker\n}\n\n// createTestFile 创建测试文件\nfunc (ts *TestSecurity) createTestFile(content string) (string, error) {\n\tfile, err := ioutil.TempFile(ts.tempDir, \"test-*.go\")\n\tif err != nil {\n\t\treturn \"\", fmt.Errorf(\"创建临时文件失败: %v\", err)\n\t}\n\tdefer file.Close()\n\n\tif _, err := file.WriteString(content); err != nil {\n\t\treturn \"\", fmt.Errorf(\"写入文件内容失败: %v\", err)\n\t}\n\n\treturn file.Name(), nil\n}\n\nfunc TestMemoryLimit(t *testing.T) {\n\tts := &TestSecurity{\n\t\ttempDir: t.TempDir(),\n\t\tdetector: detectors.NewVulnerabilityDetector(),\n\t\tchecker: utils.NewSecurityChecker(),\n\t}\n\n\t// 创建可能导致内存溢出的测试文件\n\tcontent := `\n\tpackage main\n\n\tfunc memoryIntensive() {\n\t\tlargeSlice := make([]int, 1<<30) // 尝试分配大量内存\n\t\tfor i := range largeSlice {\n\t\t\tlargeSlice[i] = i\n\t\t}\n\t}\n\t`\n\t\n\tfilePath, err := ts.createTestFile(content)\n\trequire.NoError(t, err)\n\n\t// 检查内存使用\n\tmemoryUsage, err := ts.checker.CheckMemoryUsage(filePath)\n\trequire.NoError(t, err)\n\tassert.Less(t, memoryUsage, uint64(8<<30)) // 8GB限制\n}\n\nfunc TestExecutionTimeout(t *testing.T) {\n\tts := &TestSecurity{\n\t\ttempDir: t.TempDir(),\n\t\tdetector: detectors.NewVulnerabilityDetector(),\n\t\tchecker: utils.NewSecurityChecker(),\n\t}\n\n\t// 创建可能导致无限循环的测试文件\n\tcontent := `\n\tpackage main\n\n\tfunc infiniteLoop() {\n\t\tfor {\n\t\t\t// 无限循环\n\t\t}\n\t}\n\t`\n\n\tfilePath, err := ts.createTestFile(content)\n\trequire.NoError(t, err)\n\n\t// 检查执行时间\n\terr = ts.checker.CheckExecutionTime(filePath, 5*time.Second)\n\tassert.Error(t, err)\n\tassert.Contains(t, err.Error(), \"timeout\")\n}\n\nfunc TestFileAccess(t *testing.T) {\n\tts := &TestSecurity{\n\t\ttempDir: t.TempDir(),\n\t\tdetector: detectors.NewVulnerabilityDetector(),\n\t\tchecker: utils.NewSecurityChecker(),\n\t}\n\n\t// 创建测试文件\n\tcontent := `\n\tpackage main\n\n\timport \"os\"\n\n\tfunc accessSensitiveFile() {\n\t\tfile, _ := os.Open(\"/etc/passwd\")\n\t\tdefer file.Close()\n\t}\n\t`\n\n\tfilePath, err := ts.createTestFile(content)\n\trequire.NoError(t, err)\n\n\t// 检查文件访问\n\tviolations, err := ts.checker.CheckFileAccess(filePath)\n\trequire.NoError(t, err)\n\tassert.Greater(t, len(violations), 0)\n\tassert.Contains(t, violations[0], \"/etc/passwd\")\n}\n\nfunc TestNetworkAccess(t *testing.T) {\n\tts := &TestSecurity{\n\t\ttempDir: t.TempDir(),\n\t\tdetector: detectors.NewVulnerabilityDetector(),\n\t\tchecker: utils.NewSecurityChecker(),\n\t}\n\n\t// 创建测试文件\n\tcontent := `\n\tpackage main\n\n\timport \"net\"\n\n\tfunc connectExternal() {\n\t\tconn, _ := net.Dial(\"tcp\", \"example.com:80\")\n\t\tdefer conn.Close()\n\t}\n\t`\n\n\tfilePath, err := ts.createTestFile(content)\n\trequire.NoError(t, err)\n\n\t// 检查网络访问\n\tviolations, err := ts.checker.CheckNetworkAccess(filePath)\n\trequire.NoError(t, err)\n\tassert.Greater(t, len(violations), 0)\n\tassert.Contains(t, violations[0], \"net.Dial\")\n}\n\nfunc TestCodeInjection(t *testing.T) {\n\tts := &TestSecurity{\n\t\ttempDir: t.TempDir(),\n\t\tdetector: detectors.NewVulnerabilityDetector(),\n\t\tchecker: utils.NewSecurityChecker(),\n\t}\n\n\t// 创建测试文件\n\tcontent := `\n\tpackage main\n\n\timport \"os/exec\"\n\n\tfunc executeInput(userInput string) {\n\t\tcmd := exec.Command(\"bash\", \"-c\", userInput)\n\t\tcmd.Run()\n\t}\n\t`\n\n\tfilePath, err := ts.createTestFile(content)\n\trequire.NoError(t, err)\n\n\t// 检查代码注入\n\tvulnerabilities, err := ts.detector.DetectFile(filePath)\n\trequire.NoError(t, err)\n\tassert.Greater(t, len(vulnerabilities), 0)\n\tassert.Equal(t, \"HIGH\", vulnerabilities[0].Severity)\n}\n\nfunc TestInputValidation(t *testing.T) {\n\tts := &TestSecurity{\n\t\ttempDir: t.TempDir(),\n\t\tdetector: detectors.NewVulnerabilityDetector(),\n\t\tchecker: utils.NewSecurityChecker(),\n\t}\n\n\t// 创建测试文件\n\tcontent := `\n\tpackage main\n\n\timport \"fmt\"\n\n\tfunc processInput(userInput string) {\n\t\tfmt.Sprintf(\"%s\", userInput) // 未经验证的输入\n\t}\n\t`\n\n\tfilePath, err := ts.createTestFile(content)\n\trequire.NoError(t, err)\n\n\t// 检查输入验证\n\tissues, err := ts.checker.CheckInputValidation(filePath)\n\trequire.NoError(t, err)\n\tassert.Greater(t, len(issues), 0)\n}\n\nfunc TestSecureRandom(t *testing.T) {\n\tts := &TestSecurity{\n\t\ttempDir: t.TempDir(),\n\t\tdetector: detectors.NewVulnerabilityDetector(),\n\t\tchecker: utils.NewSecurityChecker(),\n\t}\n\n\t// 创建测试文件\n\tcontent := `\n\tpackage main\n\n\timport \"math/rand\"\n\n\tfunc generateToken() string {\n\t\tconst chars = \"0123456789ABCDEF\"\n\t\tresult := make([]byte, 32)\n\t\tfor i := range result {\n\t\t\tresult[i] = chars[rand.Intn(len(chars))]\n\t\t}\n\t\treturn string(result)\n\t}\n\t`\n\n\tfilePath, err := ts.createTestFile(content)\n\trequire.NoError(t, err)\n\n\t// 检查随机数生成\n\tissues, err := ts.checker.CheckRandomGeneration(filePath)\n\trequire.NoError(t, err)\n\tassert.Greater(t, len(issues), 0)\n\tassert.Contains(t, issues[0], \"math/rand\")\n}\n\nfunc TestSensitiveData(t *testing.T) {\n\tts := &TestSecurity{\n\t\ttempDir: t.TempDir(),\n\t\tdetector: detectors.NewVulnerabilityDetector(),\n\t\tchecker: utils.NewSecurityChecker(),\n\t}\n\n\t// 创建测试文件\n\tcontent := `\n\tpackage main\n\n\timport \"fmt\"\n\n\tfunc processPassword(password string) {\n\t\tfmt.Printf(\"Password is: %s\\n\", password) // 敏感信息泄露\n\t}\n\t`\n\n\tfilePath, err := ts.createTestFile(content)\n\trequire.NoError(t, err)\n\n\t// 检查敏感数据处理\n\tissues, err := ts.checker.CheckSensitiveData(filePath)\n\trequire.NoError(t, err)\n\tassert.Greater(t, len(issues), 0)\n\tassert.Contains(t, issues[0], \"password\")\n}\n\nfunc TestSandboxEscape(t *testing.T) {\n\tts := &TestSecurity{\n\t\ttempDir: t.TempDir(),\n\t\tdetector: detectors.NewVulnerabilityDetector(),\n\t\tchecker: utils.NewSecurityChecker(),\n\t}\n\n\t// 创建测试文件\n\tcontent := `\n\tpackage main\n\n\timport (\n\t\t\"os\"\n\t\t\"os/exec\"\n\t)\n\n\tfunc dangerousOperation() {\n\t\tos.RemoveAll(\"/\")\n\t\texec.Command(\"chmod\", \"777\", \"/etc/passwd\").Run()\n\t}\n\t`\n\n\tfilePath, err := ts.createTestFile(content)\n\trequire.NoError(t, err)\n\n\t// 检查沙箱逃逸\n\tviolations, err := ts.checker.CheckSandboxEscape(filePath)\n\trequire.NoError(t, err)\n\tassert.Greater(t, len(violations), 0)\n\tassert.Contains(t, violations[0], \"os.RemoveAll\")\n} " }, { "path": "go/web/templates/report.html", "content": "\n\n\n \n \n Re-movery Vulnerability Report\n \n \n \n\n\n
\n

Re-movery Vulnerability Report

\n \n
\n
\n
\n
\n
Report Summary
\n

Generated at: {{.GeneratedAt}}

\n

Total Files Scanned: {{.TotalFiles}}

\n

Total Vulnerabilities Found: {{.TotalMatches}}

\n
\n
\n
\n
\n\n
\n
\n
\n
\n
\n
\n
\n
\n
\n
\n
\n
\n
\n
\n
\n
\n\n

Detailed Findings

\n {{range .Vulnerabilities}}\n
\n
\n
\n {{.Signature.Name}}\n \n {{.Signature.Severity}}\n \n
\n
\n
\n
ID: {{.Signature.ID}}
\n

{{.Signature.Description}}

\n \n
\n File: {{.File}}
\n Line: {{.Line}}
\n Confidence: {{printf \"%.1f%%\" (mul .Confidence 100)}}\n
\n\n
\n 
{{.Code}}
\n
\n\n {{if .Context}}\n
\n
Context:
\n
\n 
{{range .Context}}{{.}}\n{{end}}
\n
\n
\n {{end}}\n\n {{if .Signature.References}}\n
\n
References:
\n
    \n {{range .Signature.References}}\n
  • {{.}}
    • \n {{end}}\n
\n
\n {{end}}\n
\n
\n {{end}}\n
\n\n \n\n " }, { "path": "movery/__init__.py", "content": "\"\"\"\nRe-Movery - A tool for discovering modified vulnerable code clones\n\"\"\"\n\n__version__ = \"1.0.0\"\n__author__ = \"heyangxu\"\n__email__ = \"\"\n\nfrom .config.config import config\nfrom .detectors.vulnerability import VulnerabilityDetector\nfrom .utils.security import SecurityChecker\n\n__all__ = [\"config\", \"VulnerabilityDetector\", \"SecurityChecker\"] " }, { "path": "movery/analyzers/__init__.py", "content": "from .language import LanguageAnalyzer\nfrom .code_analyzer import CodeAnalyzer\n\n__all__ = ['LanguageAnalyzer', 'CodeAnalyzer'] " }, { "path": "movery/analyzers/code_analyzer.py", "content": "\"\"\"\nCode analysis utilities for Movery\n\"\"\"\nfrom typing import Dict, List, Optional\nimport os\nimport ast\nimport logging\n\nfrom movery.utils.logging import get_logger\nfrom movery.config.config import config\nfrom .language import LanguageAnalyzer, PythonAnalyzer, JavaAnalyzer, CppAnalyzer, GoAnalyzer\n\nlogger = get_logger(__name__)\n\nclass CodeAnalyzer:\n \"\"\"Code analyzer that supports multiple programming languages\"\"\"\n \n def __init__(self):\n self.analyzers = {\n \".py\": PythonAnalyzer(),\n \".java\": JavaAnalyzer(),\n \".cpp\": CppAnalyzer(),\n \".hpp\": CppAnalyzer(),\n \".cc\": CppAnalyzer(),\n \".hh\": CppAnalyzer(),\n \".go\": GoAnalyzer()\n }\n \n def analyze_file(self, filename: str) -> Dict:\n \"\"\"Analyze a source code file\"\"\"\n ext = os.path.splitext(filename)[1].lower()\n \n if ext not in self.analyzers:\n logger.warning(f\"Unsupported file type: {ext}\")\n return {\n \"complexity\": 0,\n \"functions\": [],\n \"classes\": [],\n \"imports\": [],\n \"variables\": []\n }\n \n analyzer = self.analyzers[ext]\n try:\n ast_node = analyzer.parse_file(filename)\n \n return {\n \"complexity\": self._calculate_complexity(ast_node),\n \"functions\": analyzer.get_functions(ast_node),\n \"classes\": analyzer.get_classes(ast_node),\n \"imports\": analyzer.get_imports(ast_node),\n \"variables\": analyzer.get_variables(ast_node)\n }\n \n except Exception as e:\n logger.error(f\"Error analyzing file {filename}: {str(e)}\")\n return {\n \"complexity\": 0,\n \"functions\": [],\n \"classes\": [],\n \"imports\": [],\n \"variables\": []\n }\n \n def _calculate_complexity(self, ast_node: any) -> int:\n \"\"\"Calculate code complexity\"\"\"\n # 简单实现 - 仅计算函数和类的数量\n if isinstance(ast_node, ast.AST):\n functions = sum(1 for node in ast.walk(ast_node) \n if isinstance(node, ast.FunctionDef))\n classes = sum(1 for node in ast.walk(ast_node)\n if isinstance(node, ast.ClassDef))\n return functions + classes\n return 0 " }, { "path": "movery/analyzers/language.py", "content": "\"\"\"\nLanguage analysis utilities for Movery\n\"\"\"\nimport os\nfrom typing import Dict, List, Optional, Set, Tuple, Any\nimport re\nimport ast\nimport tokenize\nfrom io import StringIO\nimport logging\nimport subprocess\nfrom abc import ABC, abstractmethod\nimport tempfile\nimport json\n\nfrom movery.config.config import config\nfrom movery.utils.logging import get_logger\nfrom movery.utils.memory import MemoryMappedFile, MemoryMonitor\n\nlogger = get_logger(__name__)\n\nclass LanguageAnalyzer(ABC):\n \"\"\"Base class for language analyzers\"\"\"\n \n def __init__(self):\n self.file_extensions = []\n \n @abstractmethod\n def parse_file(self, filename: str) -> Any:\n \"\"\"Parse source file and return AST\"\"\"\n pass\n \n @abstractmethod\n def get_functions(self, ast_node: Any) -> List[Dict]:\n \"\"\"Extract functions from AST\"\"\"\n pass\n \n @abstractmethod\n def get_classes(self, ast_node: Any) -> List[Dict]:\n \"\"\"Extract classes from AST\"\"\"\n pass\n \n @abstractmethod\n def get_imports(self, ast_node: Any) -> List[Dict]:\n \"\"\"Extract imports from AST\"\"\"\n pass\n \n @abstractmethod\n def get_variables(self, ast_node: Any) -> List[Dict]:\n \"\"\"Extract variables from AST\"\"\"\n pass\n \n def supports_file(self, filename: str) -> bool:\n \"\"\"Check if file is supported by this analyzer\"\"\"\n ext = os.path.splitext(filename)[1].lower()\n return ext in self.file_extensions\n\nclass PythonAnalyzer(LanguageAnalyzer):\n \"\"\"Python source code analyzer\"\"\"\n \n def __init__(self):\n super().__init__()\n self.file_extensions = [\".py\"]\n \n def parse_file(self, filename: str) -> ast.AST:\n \"\"\"Parse Python source file\"\"\"\n with open(filename, \"r\", encoding=\"utf-8\") as f:\n return ast.parse(f.read(), filename=filename)\n \n def get_functions(self, ast_node: ast.AST) -> List[Dict]:\n \"\"\"Extract functions from Python AST\"\"\"\n functions = []\n for node in ast.walk(ast_node):\n if isinstance(node, ast.FunctionDef):\n func = {\n \"name\": node.name,\n \"lineno\": node.lineno,\n \"args\": [arg.arg for arg in node.args.args],\n \"returns\": self._get_return_annotation(node),\n \"docstring\": ast.get_docstring(node),\n \"decorators\": [self._get_decorator_name(d) for d in node.decorator_list]\n }\n functions.append(func)\n return functions\n \n def get_classes(self, ast_node: ast.AST) -> List[Dict]:\n \"\"\"Extract classes from Python AST\"\"\"\n classes = []\n for node in ast.walk(ast_node):\n if isinstance(node, ast.ClassDef):\n cls = {\n \"name\": node.name,\n \"lineno\": node.lineno,\n \"bases\": [self._get_name(b) for b in node.bases],\n \"docstring\": ast.get_docstring(node),\n \"methods\": self.get_functions(node),\n \"decorators\": [self._get_decorator_name(d) for d in node.decorator_list]\n }\n classes.append(cls)\n return classes\n \n def get_imports(self, ast_node: ast.AST) -> List[Dict]:\n \"\"\"Extract imports from Python AST\"\"\"\n imports = []\n for node in ast.walk(ast_node):\n if isinstance(node, ast.Import):\n for name in node.names:\n imports.append({\n \"module\": name.name,\n \"alias\": name.asname,\n \"lineno\": node.lineno\n })\n elif isinstance(node, ast.ImportFrom):\n for name in node.names:\n imports.append({\n \"module\": node.module,\n \"name\": name.name,\n \"alias\": name.asname,\n \"lineno\": node.lineno\n })\n return imports\n \n def get_variables(self, ast_node: ast.AST) -> List[Dict]:\n \"\"\"Extract variables from Python AST\"\"\"\n variables = []\n for node in ast.walk(ast_node):\n if isinstance(node, ast.Assign):\n for target in node.targets:\n if isinstance(target, ast.Name):\n var = {\n \"name\": target.id,\n \"lineno\": node.lineno,\n \"value\": self._get_value(node.value)\n }\n variables.append(var)\n return variables\n \n def _get_return_annotation(self, node: ast.FunctionDef) -> Optional[str]:\n \"\"\"Get function return type annotation\"\"\"\n if node.returns:\n return self._get_name(node.returns)\n return None\n \n def _get_decorator_name(self, node: ast.expr) -> str:\n \"\"\"Get decorator name\"\"\"\n if isinstance(node, ast.Name):\n return node.id\n elif isinstance(node, ast.Call):\n return self._get_name(node.func)\n elif isinstance(node, ast.Attribute):\n return f\"{self._get_name(node.value)}.{node.attr}\"\n return str(node)\n \n def _get_name(self, node: ast.expr) -> str:\n \"\"\"Get name from AST node\"\"\"\n if isinstance(node, ast.Name):\n return node.id\n elif isinstance(node, ast.Attribute):\n return f\"{self._get_name(node.value)}.{node.attr}\"\n return str(node)\n \n def _get_value(self, node: ast.expr) -> Any:\n \"\"\"Get value from AST node\"\"\"\n if isinstance(node, (ast.Num, ast.Str, ast.Bytes)):\n return node.n if isinstance(node, ast.Num) else node.s\n elif isinstance(node, ast.NameConstant):\n return node.value\n elif isinstance(node, ast.List):\n return [self._get_value(elt) for elt in node.elts]\n elif isinstance(node, ast.Dict):\n return {self._get_value(k): self._get_value(v)\n for k, v in zip(node.keys, node.values)}\n return None\n\nclass JavaAnalyzer(LanguageAnalyzer):\n \"\"\"Java source code analyzer\"\"\"\n \n def __init__(self):\n super().__init__()\n self.file_extensions = [\".java\"]\n \n def parse_file(self, filename: str) -> Dict:\n \"\"\"Parse Java source file using external parser\"\"\"\n # Use JavaParser or similar tool\n # This is a placeholder implementation\n return {}\n \n def get_functions(self, ast_node: Dict) -> List[Dict]:\n \"\"\"Extract methods from Java AST\"\"\"\n # Placeholder implementation\n return []\n \n def get_classes(self, ast_node: Dict) -> List[Dict]:\n \"\"\"Extract classes from Java AST\"\"\"\n # Placeholder implementation\n return []\n \n def get_imports(self, ast_node: Dict) -> List[Dict]:\n \"\"\"Extract imports from Java AST\"\"\"\n # Placeholder implementation\n return []\n \n def get_variables(self, ast_node: Dict) -> List[Dict]:\n \"\"\"Extract variables from Java AST\"\"\"\n # Placeholder implementation\n return []\n\nclass CppAnalyzer(LanguageAnalyzer):\n \"\"\"C++ source code analyzer\"\"\"\n \n def __init__(self):\n super().__init__()\n self.file_extensions = [\".cpp\", \".hpp\", \".cc\", \".hh\"]\n \n def parse_file(self, filename: str) -> Dict:\n \"\"\"Parse C++ source file using external parser\"\"\"\n # Use clang or similar tool\n # This is a placeholder implementation\n return {}\n \n def get_functions(self, ast_node: Dict) -> List[Dict]:\n \"\"\"Extract functions from C++ AST\"\"\"\n # Placeholder implementation\n return []\n \n def get_classes(self, ast_node: Dict) -> List[Dict]:\n \"\"\"Extract classes from C++ AST\"\"\"\n # Placeholder implementation\n return []\n \n def get_imports(self, ast_node: Dict) -> List[Dict]:\n \"\"\"Extract includes from C++ AST\"\"\"\n # Placeholder implementation\n return []\n \n def get_variables(self, ast_node: Dict) -> List[Dict]:\n \"\"\"Extract variables from C++ AST\"\"\"\n # Placeholder implementation\n return []\n\nclass GoAnalyzer(LanguageAnalyzer):\n \"\"\"Go source code analyzer\"\"\"\n \n def __init__(self):\n super().__init__()\n self.file_extensions = [\".go\"]\n \n def parse_file(self, filename: str) -> Dict:\n \"\"\"Parse Go source file using external parser\"\"\"\n # Use go/parser or similar tool\n # This is a placeholder implementation\n return {}\n \n def get_functions(self, ast_node: Dict) -> List[Dict]:\n \"\"\"Extract functions from Go AST\"\"\"\n # Placeholder implementation\n return []\n \n def get_classes(self, ast_node: Dict) -> List[Dict]:\n \"\"\"Extract types from Go AST\"\"\"\n # Placeholder implementation\n return []\n \n def get_imports(self, ast_node: Dict) -> List[Dict]:\n \"\"\"Extract imports from Go AST\"\"\"\n # Placeholder implementation\n return []\n \n def get_variables(self, ast_node: Dict) -> List[Dict]:\n \"\"\"Extract variables from Go AST\"\"\"\n # Placeholder implementation\n return []\n\nclass JavaScriptAnalyzer(LanguageAnalyzer):\n \"\"\"JavaScript source code analyzer\"\"\"\n \n def __init__(self):\n super().__init__()\n self.file_extensions = [\".js\", \".jsx\", \".ts\", \".tsx\"]\n \n def parse_file(self, filename: str) -> Dict:\n \"\"\"Parse JavaScript source file using external parser\"\"\"\n # Use esprima or similar tool\n # This is a placeholder implementation\n return {}\n \n def get_functions(self, ast_node: Dict) -> List[Dict]:\n \"\"\"Extract functions from JavaScript AST\"\"\"\n # Placeholder implementation\n return []\n \n def get_classes(self, ast_node: Dict) -> List[Dict]:\n \"\"\"Extract classes from JavaScript AST\"\"\"\n # Placeholder implementation\n return []\n \n def get_imports(self, ast_node: Dict) -> List[Dict]:\n \"\"\"Extract imports from JavaScript AST\"\"\"\n # Placeholder implementation\n return []\n \n def get_variables(self, ast_node: Dict) -> List[Dict]:\n \"\"\"Extract variables from JavaScript AST\"\"\"\n # Placeholder implementation\n return []\n\nclass LanguageAnalyzerFactory:\n \"\"\"Factory for creating language analyzers\"\"\"\n \n _analyzers: Dict[str, LanguageAnalyzer] = {\n \"python\": PythonAnalyzer(),\n \"java\": JavaAnalyzer(),\n \"cpp\": CppAnalyzer(),\n \"go\": GoAnalyzer(),\n \"javascript\": JavaScriptAnalyzer()\n }\n \n @classmethod\n def get_analyzer(cls, filename: str) -> Optional[LanguageAnalyzer]:\n \"\"\"Get appropriate analyzer for file\"\"\"\n ext = os.path.splitext(filename)[1].lower()\n for analyzer in cls._analyzers.values():\n if analyzer.supports_file(filename):\n return analyzer\n return None\n \n @classmethod\n def register_analyzer(cls, language: str, analyzer: LanguageAnalyzer):\n \"\"\"Register new language analyzer\"\"\"\n cls._analyzers[language] = analyzer " }, { "path": "movery/config/__init__.py", "content": "\"\"\"\nConfiguration module for Movery\n\"\"\"\nfrom .config import config\n\n__all__ = ['config'] " }, { "path": "movery/config/config.json", "content": "{\n \"processing\": {\n \"num_processes\": 4,\n \"max_memory_usage\": 8589934592,\n \"chunk_size\": 1048576,\n \"enable_cache\": true,\n \"cache_dir\": \".cache\",\n \"cache_max_size\": 1073741824,\n \"supported_languages\": [\n \"c\",\n \"cpp\",\n \"java\", \n \"python\",\n \"go\",\n \"javascript\"\n ]\n },\n \"detector\": {\n \"min_similarity\": 0.8,\n \"max_edit_distance\": 10,\n \"context_lines\": 3,\n \"max_ast_depth\": 50,\n \"max_cfg_nodes\": 1000,\n \"enable_semantic_match\": true,\n \"enable_syntax_match\": true,\n \"enable_token_match\": true,\n \"report_format\": \"html\",\n \"report_dir\": \"reports\",\n \"exclude_patterns\": [\n \"**/test/*\",\n \"**/tests/*\",\n \"**/vendor/*\",\n \"**/node_modules/*\"\n ]\n },\n \"logging\": {\n \"log_level\": \"INFO\",\n \"log_file\": \"movery.log\",\n \"log_format\": \"%(asctime)s - %(name)s - %(levelname)s - %(message)s\",\n \"enable_profiling\": false,\n \"profile_output\": \"profile.stats\",\n \"show_progress\": true,\n \"progress_interval\": 1\n },\n \"security\": {\n \"max_file_size\": 104857600,\n \"allowed_schemes\": [\n \"file\",\n \"http\",\n \"https\"\n ],\n \"enable_sandbox\": true,\n \"sandbox_timeout\": 60,\n \"require_auth\": false,\n \"rate_limit\": 100,\n \"rate_limit_period\": 60\n }\n} " }, { "path": "movery/config/config.py", "content": "\"\"\"\nConfiguration module for Movery\n\"\"\"\nimport json\nimport os\nfrom typing import Dict, Any, List\nfrom dataclasses import dataclass\n\n@dataclass\nclass ProcessingConfig:\n num_processes: int\n max_memory_usage: int\n chunk_size: int\n enable_cache: bool\n cache_dir: str\n cache_max_size: int\n supported_languages: List[str]\n\n@dataclass\nclass DetectorConfig:\n min_similarity: float\n max_edit_distance: int\n context_lines: int\n max_ast_depth: int\n max_cfg_nodes: int\n enable_semantic_match: bool\n enable_syntax_match: bool\n enable_token_match: bool\n report_format: str\n report_dir: str\n exclude_patterns: List[str]\n\n@dataclass\nclass LoggingConfig:\n log_level: str\n log_file: str\n log_format: str\n enable_profiling: bool\n profile_output: str\n show_progress: bool\n progress_interval: int\n\n@dataclass\nclass SecurityConfig:\n max_file_size: int\n allowed_schemes: List[str]\n enable_sandbox: bool\n sandbox_timeout: int\n require_auth: bool\n rate_limit: int\n rate_limit_period: int\n\n@dataclass\nclass Config:\n processing: ProcessingConfig\n detector: DetectorConfig\n logging: LoggingConfig\n security: SecurityConfig\n\ndef load_config(config_path: str = None) -> Config:\n \"\"\"\n Load configuration from JSON file\n \n Args:\n config_path: Path to config file. If None, uses default config.json\n \n Returns:\n Configuration object\n \"\"\"\n if config_path is None:\n config_path = os.path.join(os.path.dirname(__file__), \"config.json\")\n \n with open(config_path, \"r\", encoding=\"utf-8\") as f:\n data = json.load(f)\n \n return Config(\n processing=ProcessingConfig(**data[\"processing\"]),\n detector=DetectorConfig(**data[\"detector\"]),\n logging=LoggingConfig(**data[\"logging\"]),\n security=SecurityConfig(**data[\"security\"])\n )\n\n# Load default configuration\nconfig = load_config() " }, { "path": "movery/config.json", "content": "{\n \"processing\": {\n \"num_processes\": 4,\n \"max_memory_usage\": 8589934592,\n \"chunk_size\": 1048576,\n \"enable_cache\": true,\n \"cache_dir\": \".cache\",\n \"cache_max_size\": 1073741824,\n \"supported_languages\": [\n \"c\",\n \"cpp\",\n \"java\", \n \"python\",\n \"go\",\n \"javascript\"\n ]\n },\n \"detector\": {\n \"min_similarity\": 0.8,\n \"max_edit_distance\": 10,\n \"context_lines\": 3,\n \"max_ast_depth\": 50,\n \"max_cfg_nodes\": 1000,\n \"enable_semantic_match\": true,\n \"enable_syntax_match\": true,\n \"enable_token_match\": true,\n \"report_format\": \"html\",\n \"report_dir\": \"reports\",\n \"exclude_patterns\": [\n \"**/test/*\",\n \"**/tests/*\",\n \"**/vendor/*\",\n \"**/node_modules/*\"\n ]\n },\n \"logging\": {\n \"log_level\": \"INFO\",\n \"log_file\": \"movery.log\",\n \"log_format\": \"%(asctime)s - %(name)s - %(levelname)s - %(message)s\",\n \"enable_profiling\": false,\n \"profile_output\": \"profile.stats\",\n \"show_progress\": true,\n \"progress_interval\": 1\n },\n \"security\": {\n \"max_file_size\": 104857600,\n \"allowed_schemes\": [\n \"file\",\n \"http\",\n \"https\"\n ],\n \"enable_sandbox\": true,\n \"sandbox_timeout\": 60,\n \"require_auth\": false,\n \"rate_limit\": 100,\n \"rate_limit_period\": 60\n }\n} " }, { "path": "movery/detectors/__init__.py", "content": "from .vulnerability import VulnerabilityDetector, Signature, VulnerabilityMatch\n\n__all__ = ['VulnerabilityDetector', 'Signature', 'VulnerabilityMatch'] " }, { "path": "movery/detectors/vulnerability.py", "content": "import json\nimport re\nfrom typing import List, Optional\nfrom dataclasses import dataclass\nimport ast\nimport astroid\nfrom abc import ABC\n\n@dataclass\nclass Signature:\n \"\"\"漏洞签名类\"\"\"\n id: str\n name: str\n severity: str\n code_patterns: List[str]\n\n@dataclass\nclass VulnerabilityMatch:\n \"\"\"漏洞匹配结果类\"\"\"\n signature: Signature\n line_number: int\n matched_code: str\n confidence: float\n\nclass VulnerabilityDetector:\n \"\"\"漏洞检测器类\"\"\"\n \n def __init__(self):\n \"\"\"初始化漏洞检测器\"\"\"\n self.signatures: List[Signature] = []\n self.min_confidence = 0.7\n\n def load_signatures(self, signature_file: str) -> None:\n \"\"\"从JSON文件加载漏洞签名\n\n Args:\n signature_file: 签名文件路径\n \"\"\"\n with open(signature_file, 'r') as f:\n data = json.load(f)\n self.signatures = [\n Signature(\n id=sig['id'],\n name=sig['name'],\n severity=sig['severity'],\n code_patterns=sig['code_patterns']\n )\n for sig in data['signatures']\n ]\n\n def detect_file(self, file_path: str) -> List[VulnerabilityMatch]:\n \"\"\"检测文件中的漏洞\n\n Args:\n file_path: 待检测的文件路径\n\n Returns:\n List[VulnerabilityMatch]: 漏洞匹配结果列表\n \"\"\"\n with open(file_path, 'r') as f:\n content = f.read()\n \n matches = []\n for signature in self.signatures:\n for pattern in signature.code_patterns:\n # 使用正则表达式查找匹配\n for match in re.finditer(pattern, content):\n line_number = content.count('\\n', 0, match.start()) + 1\n matched_code = match.group()\n confidence = self._calculate_confidence(matched_code, pattern)\n \n if confidence >= self.min_confidence:\n matches.append(VulnerabilityMatch(\n signature=signature,\n line_number=line_number,\n matched_code=matched_code,\n confidence=confidence\n ))\n\n return matches\n\n def _calculate_confidence(self, matched_code: str, pattern: str) -> float:\n \"\"\"计算匹配的置信度\n\n Args:\n matched_code: 匹配到的代码\n pattern: 匹配模式\n\n Returns:\n float: 置信度(0-1)\n \"\"\"\n # 基本匹配的置信度为0.8\n confidence = 0.8\n \n # 根据匹配的完整性增加置信度\n if len(matched_code) > 10:\n confidence += 0.05\n \n # 根据上下文增加置信度\n if 'import' in matched_code:\n confidence += 0.05\n \n # 根据模式的特异性增加置信度\n if len(pattern) > 20:\n confidence += 0.05\n \n # 根据函数调用的参数增加置信度\n if '(' in matched_code and ')' in matched_code:\n confidence += 0.05\n \n return round(min(confidence, 1.0), 2)\n\n def analyze_ast(self, node: ast.AST) -> List[VulnerabilityMatch]:\n \"\"\"分析AST节点中的漏洞\n\n Args:\n node: AST节点\n\n Returns:\n List[VulnerabilityMatch]: 漏洞匹配结果列表\n \"\"\"\n matches = []\n \n # 遍历AST节点\n for child in ast.walk(node):\n # 检查函数调用\n if isinstance(child, ast.Call):\n func = child.func\n func_name = None\n \n if isinstance(func, ast.Name):\n func_name = func.id\n elif isinstance(func, ast.Attribute):\n try:\n parts = []\n current = func\n while isinstance(current, ast.Attribute):\n parts.append(current.attr)\n current = current.value\n if isinstance(current, ast.Name):\n parts.append(current.id)\n func_name = \".\".join(reversed(parts))\n except AttributeError:\n continue\n \n if func_name is None:\n continue\n \n # 获取函数调用的参数\n args_str = \"\"\n for arg in child.args:\n if isinstance(arg, ast.Constant):\n args_str += str(arg.value)\n elif isinstance(arg, ast.Name):\n args_str += arg.id\n elif isinstance(arg, ast.List):\n args_str += \"[...]\"\n elif isinstance(arg, ast.Dict):\n args_str += \"{...}\"\n \n # 构建完整的函数调用字符串\n full_call = f\"{func_name}({args_str})\"\n \n # 检查是否匹配任何签名\n for signature in self.signatures:\n for pattern in signature.code_patterns:\n try:\n if re.search(pattern, full_call, re.IGNORECASE):\n confidence = self._calculate_confidence(full_call, pattern)\n if confidence >= self.min_confidence:\n matches.append(VulnerabilityMatch(\n signature=signature,\n line_number=child.lineno,\n matched_code=full_call,\n confidence=confidence\n ))\n except re.error:\n continue\n \n return matches\n\n def detect_similar_patterns(self, code: str, threshold: float = 0.7) -> List[VulnerabilityMatch]:\n \"\"\"检测相似的漏洞模式\n\n Args:\n code: 待检测的代码\n threshold: 相似度阈值\n\n Returns:\n List[VulnerabilityMatch]: 漏洞匹配结果列表\n \"\"\"\n matches = []\n \n # 使用astroid进行更深入的代码分析\n try:\n module = astroid.parse(code)\n for node in module.nodes_of_class(astroid.Call):\n # 获取调用的函数名和参数\n func_name = None\n args_str = \"\"\n \n try:\n if isinstance(node.func, astroid.Name):\n func_name = node.func.name\n elif isinstance(node.func, astroid.Attribute):\n parts = []\n current = node.func\n while isinstance(current, astroid.Attribute):\n parts.append(current.attrname)\n current = current.expr\n if isinstance(current, astroid.Name):\n parts.append(current.name)\n func_name = \".\".join(reversed(parts))\n \n # 获取参数\n for arg in node.args:\n if isinstance(arg, astroid.Const):\n args_str += str(arg.value)\n elif isinstance(arg, astroid.Name):\n args_str += arg.name\n else:\n args_str += \"...\"\n args_str += \", \"\n args_str = args_str.rstrip(\", \")\n \n except (AttributeError, astroid.AstroidError):\n continue\n \n if func_name is None:\n continue\n \n # 构建完整的函数调用字符串\n full_call = f\"{func_name}({args_str})\"\n \n # 检查每个签名\n for signature in self.signatures:\n for pattern in signature.code_patterns:\n try:\n # 首先尝试正则表达式匹配\n if re.search(pattern, full_call, re.IGNORECASE):\n confidence = self._calculate_confidence(full_call, pattern)\n matches.append(VulnerabilityMatch(\n signature=signature,\n line_number=node.lineno,\n matched_code=full_call,\n confidence=confidence\n ))\n else:\n # 如果正则表达式不匹配，尝试计算相似度\n similarity = self._calculate_similarity(full_call, pattern)\n if similarity >= threshold:\n matches.append(VulnerabilityMatch(\n signature=signature,\n line_number=node.lineno,\n matched_code=full_call,\n confidence=similarity\n ))\n except (re.error, Exception):\n continue\n \n except astroid.AstroidError:\n pass # 忽略解析错误\n \n return matches\n\n def _calculate_similarity(self, str1: str, str2: str) -> float:\n \"\"\"计算两个字符串的相似度\n\n Args:\n str1: 第一个字符串\n str2: 第二个字符串\n\n Returns:\n float: 相似度(0-1)\n \"\"\"\n # 使用最长公共子序列(LCS)计算相似度\n m, n = len(str1), len(str2)\n dp = [[0] * (n + 1) for _ in range(m + 1)]\n \n for i in range(1, m + 1):\n for j in range(1, n + 1):\n if str1[i-1] == str2[j-1]:\n dp[i][j] = dp[i-1][j-1] + 1\n else:\n dp[i][j] = max(dp[i-1][j], dp[i][j-1])\n \n lcs_length = dp[m][n]\n return lcs_length / max(m, n) # 归一化的相似度\n" }, { "path": "movery/go/cmd/movery/main.go", "content": "" }, { "path": "movery/main.py", "content": "\"\"\"\nMain entry point for Movery\n\"\"\"\nimport os\nimport sys\nimport argparse\nimport logging\nimport json\nfrom typing import List, Dict, Optional\nimport time\nfrom pathlib import Path\nimport concurrent.futures\nimport traceback\n\nfrom movery.config.config import config, MoveryConfig\nfrom movery.utils.logging import setup_logging, get_logger\nfrom movery.utils.memory import memory_monitor\nfrom movery.utils.parallel import worker_pool\nfrom movery.analyzers.language import LanguageAnalyzerFactory\nfrom movery.detectors.vulnerability import detector\nfrom movery.reporters.html import reporter\n\nlogger = get_logger(__name__)\n\ndef parse_args():\n \"\"\"Parse command line arguments\"\"\"\n parser = argparse.ArgumentParser(\n description=\"Movery - A tool for discovering modified vulnerable code clones\"\n )\n \n parser.add_argument(\n \"target\",\n help=\"Target program or directory to analyze\"\n )\n \n parser.add_argument(\n \"-c\", \"--config\",\n help=\"Path to configuration file\",\n default=\"config.json\"\n )\n \n parser.add_argument(\n \"-s\", \"--signatures\",\n help=\"Path to vulnerability signatures file\",\n default=\"signatures.json\"\n )\n \n parser.add_argument(\n \"-o\", \"--output\",\n help=\"Output directory for reports\",\n default=\"reports\"\n )\n \n parser.add_argument(\n \"-j\", \"--jobs\",\n help=\"Number of parallel jobs\",\n type=int,\n default=None\n )\n \n parser.add_argument(\n \"-v\", \"--verbose\",\n help=\"Enable verbose output\",\n action=\"store_true\"\n )\n \n parser.add_argument(\n \"--cache\",\n help=\"Enable result caching\",\n action=\"store_true\"\n )\n \n return parser.parse_args()\n\ndef load_config(config_file: str) -> MoveryConfig:\n \"\"\"Load configuration from file\"\"\"\n if os.path.exists(config_file):\n return MoveryConfig.from_file(config_file)\n return MoveryConfig()\n\ndef find_source_files(target: str) -> List[str]:\n \"\"\"Find all source files in target\"\"\"\n source_files = []\n \n for root, _, files in os.walk(target):\n for file in files:\n file_path = os.path.join(root, file)\n \n # Skip files larger than limit\n if os.path.getsize(file_path) > config.security.max_file_size:\n logger.warning(f\"Skipping large file: {file_path}\")\n continue\n \n # Skip files matching exclude patterns\n skip = False\n for pattern in config.detector.exclude_patterns:\n if Path(file_path).match(pattern):\n skip = True\n break\n if skip:\n continue\n \n # Check if file is supported\n if LanguageAnalyzerFactory.get_analyzer(file_path):\n source_files.append(file_path)\n \n return source_files\n\ndef analyze_file(file: str) -> List[Dict]:\n \"\"\"Analyze single file for vulnerabilities\"\"\"\n try:\n matches = detector.detect(file)\n return [match.to_dict() for match in matches]\n except Exception as e:\n logger.error(f\"Error analyzing file {file}: {str(e)}\")\n logger.debug(traceback.format_exc())\n return []\n\ndef main():\n \"\"\"Main entry point\"\"\"\n start_time = time.time()\n \n # Parse arguments\n args = parse_args()\n \n # Setup logging\n log_level = logging.DEBUG if args.verbose else logging.INFO\n setup_logging(level=log_level)\n \n logger.info(\"Starting Movery...\")\n \n try:\n # Load configuration\n config = load_config(args.config)\n if args.jobs:\n config.processing.num_processes = args.jobs\n config.processing.enable_cache = args.cache\n \n # Load vulnerability signatures\n detector.load_signatures(args.signatures)\n \n # Find source files\n target_path = os.path.abspath(args.target)\n if not os.path.exists(target_path):\n raise FileNotFoundError(f\"Target not found: {target_path}\")\n \n logger.info(f\"Analyzing target: {target_path}\")\n source_files = find_source_files(target_path)\n logger.info(f\"Found {len(source_files)} source files\")\n \n # Start worker pool\n worker_pool.start()\n \n # Process files in parallel\n all_matches = []\n with concurrent.futures.ThreadPoolExecutor(\n max_workers=config.processing.num_processes\n ) as executor:\n future_to_file = {\n executor.submit(analyze_file, file): file\n for file in source_files\n }\n \n for future in concurrent.futures.as_completed(future_to_file):\n file = future_to_file[future]\n try:\n matches = future.result()\n if matches:\n all_matches.extend(matches)\n logger.info(\n f\"Found {len(matches)} vulnerabilities in {file}\")\n except Exception as e:\n logger.error(f\"Error processing {file}: {str(e)}\")\n \n # Generate report\n if all_matches:\n os.makedirs(args.output, exist_ok=True)\n report_file = os.path.join(\n args.output,\n f\"report_{int(time.time())}.html\"\n )\n reporter.generate_report(all_matches, report_file)\n logger.info(f\"Generated report: {report_file}\")\n else:\n logger.info(\"No vulnerabilities found\")\n \n elapsed_time = time.time() - start_time\n logger.info(f\"Analysis completed in {elapsed_time:.2f} seconds\")\n \n except Exception as e:\n logger.error(f\"Error: {str(e)}\")\n logger.debug(traceback.format_exc())\n sys.exit(1)\n finally:\n worker_pool.stop()\n\nif __name__ == \"__main__\":\n main() " }, { "path": "movery/reporters/__init__.py", "content": "from .html import HTMLReporter\n\n__all__ = ['HTMLReporter'] " }, { "path": "movery/reporters/html.py", "content": "\"\"\"\nHTML report generator for Movery\n\"\"\"\nimport os\nfrom typing import List, Dict, Any\nimport json\nimport datetime\nfrom jinja2 import Environment, FileSystemLoader\nimport logging\nimport base64\nimport plotly.graph_objects as go\nimport plotly.express as px\nimport pandas as pd\n\nfrom movery.config.config import config\nfrom movery.utils.logging import get_logger\nfrom movery.detectors.vulnerability import VulnerabilityMatch\n\nlogger = get_logger(__name__)\n\nclass HTMLReporter:\n \"\"\"Generate HTML vulnerability reports\"\"\"\n \n def __init__(self, template_dir: str = \"templates\"):\n self.template_dir = template_dir\n self.env = Environment(loader=FileSystemLoader(template_dir))\n \n def generate_report(self, matches: List[VulnerabilityMatch],\n output_file: str):\n \"\"\"Generate HTML report from vulnerability matches\"\"\"\n template = self.env.get_template(\"report.html\")\n \n # Prepare report data\n report_data = self._prepare_report_data(matches)\n \n # Generate charts\n charts = self._generate_charts(matches)\n \n # Render template\n html = template.render(\n report=report_data,\n charts=charts,\n generated_at=datetime.datetime.now().isoformat()\n )\n \n # Write report\n os.makedirs(os.path.dirname(output_file), exist_ok=True)\n with open(output_file, \"w\", encoding=\"utf-8\") as f:\n f.write(html)\n \n logger.info(f\"Generated HTML report: {output_file}\")\n \n def _prepare_report_data(self, matches: List[VulnerabilityMatch]) -> Dict:\n \"\"\"Prepare report data from matches\"\"\"\n vulnerabilities = []\n files = set()\n severities = {\"CRITICAL\": 0, \"HIGH\": 0, \"MEDIUM\": 0, \"LOW\": 0}\n \n for match in matches:\n vulnerabilities.append({\n \"id\": match.signature.id,\n \"name\": match.signature.name,\n \"description\": match.signature.description,\n \"severity\": match.signature.severity,\n \"cwe_id\": match.signature.cwe_id,\n \"cve_id\": match.signature.cve_id,\n \"file\": match.file,\n \"line_start\": match.line_start,\n \"line_end\": match.line_end,\n \"matched_code\": match.matched_code,\n \"confidence\": match.confidence,\n \"context\": match.context\n })\n \n files.add(match.file)\n severities[match.signature.severity] = \\\n severities.get(match.signature.severity, 0) + 1\n \n return {\n \"summary\": {\n \"total_vulnerabilities\": len(vulnerabilities),\n \"total_files\": len(files),\n \"severities\": severities\n },\n \"vulnerabilities\": vulnerabilities,\n \"files\": sorted(list(files))\n }\n \n def _generate_charts(self, matches: List[VulnerabilityMatch]) -> Dict[str, str]:\n \"\"\"Generate charts for report\"\"\"\n charts = {}\n \n # Severity distribution pie chart\n severity_counts = pd.DataFrame([\n {\"severity\": m.signature.severity, \"count\": 1}\n for m in matches\n ]).groupby(\"severity\").sum().reset_index()\n \n fig = px.pie(severity_counts, values=\"count\", names=\"severity\",\n title=\"Vulnerability Severity Distribution\")\n charts[\"severity_distribution\"] = self._fig_to_base64(fig)\n \n # Vulnerability types bar chart\n vuln_types = pd.DataFrame([\n {\"type\": m.signature.name, \"count\": 1}\n for m in matches\n ]).groupby(\"type\").sum().reset_index()\n \n fig = px.bar(vuln_types, x=\"type\", y=\"count\",\n title=\"Vulnerability Types\")\n fig.update_layout(xaxis_tickangle=-45)\n charts[\"vulnerability_types\"] = self._fig_to_base64(fig)\n \n # Files with most vulnerabilities\n file_counts = pd.DataFrame([\n {\"file\": m.file, \"count\": 1}\n for m in matches\n ]).groupby(\"file\").sum().reset_index()\n \n file_counts = file_counts.sort_values(\"count\", ascending=False).head(10)\n \n fig = px.bar(file_counts, x=\"file\", y=\"count\",\n title=\"Files with Most Vulnerabilities\")\n fig.update_layout(xaxis_tickangle=-45)\n charts[\"file_distribution\"] = self._fig_to_base64(fig)\n \n # Confidence distribution histogram\n confidence_data = pd.DataFrame([\n {\"confidence\": m.confidence}\n for m in matches\n ])\n \n fig = px.histogram(confidence_data, x=\"confidence\",\n title=\"Detection Confidence Distribution\")\n charts[\"confidence_distribution\"] = self._fig_to_base64(fig)\n \n return charts\n \n def _fig_to_base64(self, fig: go.Figure) -> str:\n \"\"\"Convert plotly figure to base64 string\"\"\"\n img_bytes = fig.to_image(format=\"png\")\n return base64.b64encode(img_bytes).decode()\n\n# HTML report template\nREPORT_TEMPLATE = \"\"\"\n\n\n\n Movery Vulnerability Report\n \n\n\n
\n

Movery Vulnerability Report

\n \n
\n

Summary

\n

Total Vulnerabilities: {{ report.summary.total_vulnerabilities }}

\n

Total Files: {{ report.summary.total_files }}

\n

Severity Distribution:

\n
    \n {% for severity, count in report.summary.severities.items() %}\n
  • {{ severity }}: {{ count }}
    • \n {% endfor %}\n
\n
\n \n
\n
\n \"Severity\n
\n
\n \"Vulnerability\n
\n
\n \"File\n
\n
\n \"Confidence\n
\n
\n \n

Vulnerabilities

\n {% for vuln in report.vulnerabilities %}\n
\n
\n

{{ vuln.name }}

\n {{ vuln.severity }}\n
\n \n

{{ vuln.description }}

\n \n {% if vuln.cwe_id %}\n

CWE: {{ vuln.cwe_id }}

\n {% endif %}\n \n {% if vuln.cve_id %}\n

CVE: {{ vuln.cve_id }}

\n {% endif %}\n \n

File: {{ vuln.file }}:{{ vuln.line_start }}-{{ vuln.line_end }}

\n

Confidence: {{ \"%.2f\"|format(vuln.confidence) }}

\n \n
{{ vuln.matched_code }}
\n \n
\n

Context

\n

Imports: {{ vuln.context.imports|length }}

\n

Functions: {{ vuln.context.functions|length }}

\n

Classes: {{ vuln.context.classes|length }}

\n

Variables: {{ vuln.context.variables|length }}

\n
\n
\n {% endfor %}\n \n
\n Generated at {{ generated_at }}\n
\n
\n\n\n\"\"\"\n\n# Create templates directory and save template\nos.makedirs(\"templates\", exist_ok=True)\nwith open(\"templates/report.html\", \"w\", encoding=\"utf-8\") as f:\n f.write(REPORT_TEMPLATE)\n\n# Global reporter instance\nreporter = HTMLReporter() " }, { "path": "movery/templates/report.html", "content": "\n\n\n\n Movery Vulnerability Report\n \n\n\n
\n

Movery Vulnerability Report

\n \n
\n

Summary

\n

Total Vulnerabilities: {{ report.summary.total_vulnerabilities }}

\n

Total Files: {{ report.summary.total_files }}

\n

Severity Distribution:

\n
    \n {% for severity, count in report.summary.severities.items() %}\n
  • {{ severity }}: {{ count }}
    • \n {% endfor %}\n
\n
\n \n
\n
\n \"Severity\n
\n
\n \"Vulnerability\n
\n
\n \"File\n
\n
\n \"Confidence\n
\n
\n \n

Vulnerabilities

\n {% for vuln in report.vulnerabilities %}\n
\n
\n

{{ vuln.name }}

\n {{ vuln.severity }}\n
\n \n

{{ vuln.description }}

\n \n {% if vuln.cwe_id %}\n

CWE: {{ vuln.cwe_id }}

\n {% endif %}\n \n {% if vuln.cve_id %}\n

CVE: {{ vuln.cve_id }}

\n {% endif %}\n \n

File: {{ vuln.file }}:{{ vuln.line_start }}-{{ vuln.line_end }}

\n

Confidence: {{ \"%.2f\"|format(vuln.confidence) }}

\n \n
{{ vuln.matched_code }}
\n \n
\n

Context

\n

Imports: {{ vuln.context.imports|length }}

\n

Functions: {{ vuln.context.functions|length }}

\n

Classes: {{ vuln.context.classes|length }}

\n

Variables: {{ vuln.context.variables|length }}

\n
\n
\n {% endfor %}\n \n
\n Generated at {{ generated_at }}\n
\n
\n\n\n" }, { "path": "movery/tests/integration/test_workflow.py", "content": "import unittest\nimport os\nimport json\nimport tempfile\nimport shutil\nimport time\nfrom movery.detectors.vulnerability import VulnerabilityDetector\nfrom movery.utils.security import SecurityChecker\nfrom movery.analyzers.code_analyzer import CodeAnalyzer\nfrom movery.reporters.html import HTMLReporter\n\nclass TestWorkflow(unittest.TestCase):\n def setUp(self):\n \"\"\"测试前的准备工作\"\"\"\n self.test_dir = tempfile.mkdtemp()\n self.create_test_project()\n \n # 初始化组件\n self.detector = VulnerabilityDetector()\n self.checker = SecurityChecker()\n self.analyzer = CodeAnalyzer()\n self.reporter = HTMLReporter()\n\n def create_test_project(self):\n \"\"\"创建测试项目结构\"\"\"\n # 创建配置文件\n config = {\n \"project_name\": \"Test Project\",\n \"scan_paths\": [\"src\"],\n \"exclude_paths\": [\"tests\", \"docs\"],\n \"report_format\": \"html\",\n \"report_path\": \"reports\",\n \"severity_threshold\": \"medium\",\n \"parallel_processing\": True,\n \"max_workers\": 4\n }\n \n config_file = os.path.join(self.test_dir, \"config.json\")\n with open(config_file, \"w\") as f:\n json.dump(config, f, indent=4)\n \n # 创建签名文件\n signatures = {\n \"signatures\": [\n {\n \"id\": \"CMD001\",\n \"name\": \"命令注入\",\n \"severity\": \"high\",\n \"code_patterns\": [\n \"os\\\\.system\\\\([^)]*\\\\)\",\n \"subprocess\\\\.call\\\\([^)]*\\\\)\"\n ]\n },\n {\n \"id\": \"SQL001\",\n \"name\": \"SQL注入\",\n \"severity\": \"high\",\n \"code_patterns\": [\n \"execute\\\\(['\\\"][^'\\\"]*%[^'\\\"]*['\\\"]\\\\)\",\n \"executemany\\\\(['\\\"][^'\\\"]*%[^'\\\"]*['\\\"]\\\\)\"\n ]\n }\n ]\n }\n \n signatures_file = os.path.join(self.test_dir, \"signatures.json\")\n with open(signatures_file, \"w\") as f:\n json.dump(signatures, f, indent=4)\n \n # 创建源代码目录\n src_dir = os.path.join(self.test_dir, \"src\")\n os.makedirs(src_dir)\n \n # 创建测试源代码文件\n vulnerable_code = '''\nimport os\nimport subprocess\nimport sqlite3\n\ndef process_command(cmd):\n # 命令注入漏洞\n os.system(cmd)\n subprocess.call(cmd, shell=True)\n\ndef query_database(user_id):\n # SQL注入漏洞\n conn = sqlite3.connect(\"test.db\")\n cursor = conn.cursor()\n cursor.execute(\"SELECT * FROM users WHERE id = %s\" % user_id)\n return cursor.fetchall()\n\ndef unsafe_file_operations():\n # 不安全的文件操作\n with open(\"/etc/passwd\", \"r\") as f:\n data = f.read()\n return data\n\ndef main():\n # 调用不安全的函数\n process_command(\"ls -l\")\n query_database(\"1 OR 1=1\")\n unsafe_file_operations()\n\nif __name__ == \"__main__\":\n main()\n'''\n \n vulnerable_file = os.path.join(src_dir, \"vulnerable.py\")\n with open(vulnerable_file, \"w\") as f:\n f.write(vulnerable_code)\n \n safe_code = '''\nimport subprocess\nimport sqlite3\n\ndef safe_command(cmd):\n # 安全的命令执行\n allowed_commands = [\"ls\", \"pwd\", \"echo\"]\n if cmd.split()[0] not in allowed_commands:\n raise ValueError(\"Command not allowed\")\n subprocess.run(cmd.split(), check=True)\n\ndef safe_query(user_id):\n # 安全的数据库查询\n conn = sqlite3.connect(\"test.db\")\n cursor = conn.cursor()\n cursor.execute(\"SELECT * FROM users WHERE id = ?\", (user_id,))\n return cursor.fetchall()\n\ndef safe_file_operations():\n # 安全的文件操作\n try:\n with open(\"data.txt\", \"r\") as f:\n data = f.read()\n return data\n except Exception as e:\n return str(e)\n\ndef main():\n # 调用安全的函数\n safe_command(\"ls -l\")\n safe_query(1)\n safe_file_operations()\n\nif __name__ == \"__main__\":\n main()\n'''\n \n safe_file = os.path.join(src_dir, \"safe.py\")\n with open(safe_file, \"w\") as f:\n f.write(safe_code)\n \n # 创建报告目录\n report_dir = os.path.join(self.test_dir, \"reports\")\n os.makedirs(report_dir)\n\n def tearDown(self):\n \"\"\"测试后的清理工作\"\"\"\n shutil.rmtree(self.test_dir)\n\n def test_full_workflow(self):\n \"\"\"测试完整工作流程\"\"\"\n # 加载配置\n config_file = os.path.join(self.test_dir, \"config.json\")\n with open(config_file, \"r\") as f:\n config = json.load(f)\n \n # 加载签名\n signatures_file = os.path.join(self.test_dir, \"signatures.json\")\n self.detector.load_signatures(signatures_file)\n \n # 分析源代码文件\n src_dir = os.path.join(self.test_dir, \"src\")\n vulnerable_file = os.path.join(src_dir, \"vulnerable.py\")\n safe_file = os.path.join(src_dir, \"safe.py\")\n \n # 检测漏洞\n vulnerable_matches = self.detector.detect_file(vulnerable_file)\n safe_matches = self.detector.detect_file(safe_file)\n \n self.assertGreater(len(vulnerable_matches), 0)\n self.assertEqual(len(safe_matches), 0)\n \n # 执行安全检查\n vulnerable_security = self.checker.perform_full_check(vulnerable_file)\n safe_security = self.checker.perform_full_check(safe_file)\n \n self.assertTrue(any(result[\"has_issues\"] for result in vulnerable_security.values()))\n self.assertFalse(any(result[\"has_issues\"] for result in safe_security.values()))\n \n # 代码分析\n vulnerable_analysis = self.analyzer.analyze_file(vulnerable_file)\n safe_analysis = self.analyzer.analyze_file(safe_file)\n \n self.assertGreater(vulnerable_analysis[\"complexity\"], safe_analysis[\"complexity\"])\n \n # 生成报告\n report_data = {\n \"project_name\": config[\"project_name\"],\n \"scan_time\": time.strftime(\"%Y-%m-%d %H:%M:%S\"),\n \"files_scanned\": [vulnerable_file, safe_file],\n \"vulnerability_results\": {\n \"vulnerable.py\": vulnerable_matches,\n \"safe.py\": safe_matches\n },\n \"security_results\": {\n \"vulnerable.py\": vulnerable_security,\n \"safe.py\": safe_security\n },\n \"analysis_results\": {\n \"vulnerable.py\": vulnerable_analysis,\n \"safe.py\": safe_analysis\n }\n }\n \n report_file = os.path.join(self.test_dir, \"reports\", \"report.html\")\n self.reporter.generate_report(report_data, report_file)\n \n self.assertTrue(os.path.exists(report_file))\n self.assertGreater(os.path.getsize(report_file), 0)\n\n def test_parallel_processing(self):\n \"\"\"测试并行处理功能\"\"\"\n # 创建多个测试文件\n src_dir = os.path.join(self.test_dir, \"src\")\n test_files = []\n \n for i in range(5):\n file_path = os.path.join(src_dir, f\"test_{i}.py\")\n with open(file_path, \"w\") as f:\n f.write(\"import os\\nos.system('ls')\\n\")\n test_files.append(file_path)\n \n # 串行处理时间\n start_time = time.time()\n for file_path in test_files:\n self.detector.detect_file(file_path)\n self.checker.perform_full_check(file_path)\n self.analyzer.analyze_file(file_path)\n serial_time = time.time() - start_time\n \n # 并行处理时间\n start_time = time.time()\n import concurrent.futures\n with concurrent.futures.ThreadPoolExecutor(max_workers=4) as executor:\n futures = []\n for file_path in test_files:\n futures.append(executor.submit(self.detector.detect_file, file_path))\n futures.append(executor.submit(self.checker.perform_full_check, file_path))\n futures.append(executor.submit(self.analyzer.analyze_file, file_path))\n concurrent.futures.wait(futures)\n parallel_time = time.time() - start_time\n \n self.assertLess(parallel_time, serial_time)\n\n def test_error_handling(self):\n \"\"\"测试错误处理\"\"\"\n # 测试无效的配置文件\n invalid_config = os.path.join(self.test_dir, \"invalid_config.json\")\n with open(invalid_config, \"w\") as f:\n f.write(\"invalid json\")\n \n with self.assertRaises(json.JSONDecodeError):\n with open(invalid_config, \"r\") as f:\n json.load(f)\n \n # 测试不存在的源代码文件\n non_existent_file = os.path.join(self.test_dir, \"non_existent.py\")\n \n with self.assertRaises(FileNotFoundError):\n self.detector.detect_file(non_existent_file)\n \n # 测试无效的源代码\n invalid_code = os.path.join(self.test_dir, \"invalid_code.py\")\n with open(invalid_code, \"w\") as f:\n f.write(\"invalid python code\")\n \n with self.assertRaises(SyntaxError):\n self.analyzer.analyze_file(invalid_code)\n\nif __name__ == '__main__':\n unittest.main() " }, { "path": "movery/tests/security/test_security.py", "content": "import unittest\nimport os\nimport sys\nimport tempfile\nimport shutil\nimport subprocess\nsys.path.append(os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__)))))\n\nfrom movery.detectors.vulnerability import VulnerabilityDetector\nfrom movery.utils.security import SecurityChecker\n\nclass TestSecurity(unittest.TestCase):\n def setUp(self):\n \"\"\"设置测试环境\"\"\"\n self.test_dir = tempfile.mkdtemp()\n self.security_checker = SecurityChecker()\n self.detector = VulnerabilityDetector()\n\n def create_test_file(self, content):\n \"\"\"创建测试文件\"\"\"\n file_path = os.path.join(self.test_dir, 'test_file.py')\n with open(file_path, 'w') as f:\n f.write(content)\n return file_path\n\n def test_memory_limit(self):\n \"\"\"测试内存限制\"\"\"\n # 创建一个可能导致内存溢出的文件\n test_file = self.create_test_file('''\n def memory_intensive():\n large_list = [i for i in range(10**8)] # 尝试创建大列表\n return large_list\n ''')\n\n # 检查内存使用\n memory_usage = self.security_checker.check_memory_usage(test_file)\n self.assertLess(memory_usage, 8 * 1024 * 1024 * 1024) # 8GB限制\n\n def test_execution_timeout(self):\n \"\"\"测试执行超时\"\"\"\n # 创建一个可能导致无限循环的文件\n test_file = self.create_test_file('''\n def infinite_loop():\n while True:\n pass\n ''')\n\n # 检查执行时间\n with self.assertRaises(TimeoutError):\n self.security_checker.check_execution_time(test_file, timeout=5)\n\n def test_file_access(self):\n \"\"\"测试文件访问限制\"\"\"\n # 创建测试文件\n test_file = self.create_test_file('''\n import os\n\n def access_sensitive_file():\n with open('/etc/passwd', 'r') as f:\n return f.read()\n ''')\n\n # 检查文件访问\n violations = self.security_checker.check_file_access(test_file)\n self.assertTrue(len(violations) > 0)\n self.assertIn('/etc/passwd', violations[0])\n\n def test_network_access(self):\n \"\"\"测试网络访问限制\"\"\"\n # 创建测试文件\n test_file = self.create_test_file('''\n import socket\n\n def connect_external():\n sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)\n sock.connect(('example.com', 80))\n ''')\n\n # 检查网络访问\n violations = self.security_checker.check_network_access(test_file)\n self.assertTrue(len(violations) > 0)\n self.assertIn('socket.connect', violations[0])\n\n def test_code_injection(self):\n \"\"\"测试代码注入防护\"\"\"\n # 创建测试文件\n test_file = self.create_test_file('''\n def execute_input(user_input):\n exec(user_input) # 危险的代码执行\n ''')\n\n # 检查代码注入\n vulnerabilities = self.detector.detect_file(test_file)\n self.assertTrue(len(vulnerabilities) > 0)\n self.assertEqual(vulnerabilities[0].severity, 'HIGH')\n\n def test_input_validation(self):\n \"\"\"测试输入验证\"\"\"\n # 创建测试文件\n test_file = self.create_test_file('''\n def process_input(user_input):\n # 没有验证的输入处理\n return eval(user_input)\n ''')\n\n # 检查输入验证\n issues = self.security_checker.check_input_validation(test_file)\n self.assertTrue(len(issues) > 0)\n self.assertIn('eval', str(issues[0]))\n\n def test_secure_random(self):\n \"\"\"测试安全随机数生成\"\"\"\n # 创建测试文件\n test_file = self.create_test_file('''\n import random\n\n def generate_token():\n return ''.join(random.choice('0123456789ABCDEF') for i in range(32))\n ''')\n\n # 检查随机数生成\n issues = self.security_checker.check_random_generation(test_file)\n self.assertTrue(len(issues) > 0)\n self.assertIn('random.choice', str(issues[0]))\n\n def test_sensitive_data(self):\n \"\"\"测试敏感数据处理\"\"\"\n # 创建测试文件\n test_file = self.create_test_file('''\n def process_password(password):\n print(f\"Password is: {password}\") # 敏感信息泄露\n return hash(password) # 不安全的哈希\n ''')\n\n # 检查敏感数据处理\n issues = self.security_checker.check_sensitive_data(test_file)\n self.assertTrue(len(issues) > 0)\n self.assertIn('password', str(issues[0]).lower())\n\n def test_sandbox_escape(self):\n \"\"\"测试沙箱逃逸防护\"\"\"\n # 创建测试文件\n test_file = self.create_test_file('''\n import subprocess\n import os\n\n def dangerous_operation():\n os.system('rm -rf /') # 危险的系统命令\n subprocess.call(['chmod', '777', '/etc/passwd']) # 危险的权限修改\n ''')\n\n # 检查沙箱逃逸\n violations = self.security_checker.check_sandbox_escape(test_file)\n self.assertTrue(len(violations) > 0)\n self.assertIn('os.system', str(violations[0]))\n\n def tearDown(self):\n \"\"\"清理测试环境\"\"\"\n shutil.rmtree(self.test_dir)\n\nif __name__ == '__main__':\n unittest.main() " }, { "path": "movery/tests/unit/test_analyzer.py", "content": "import unittest\nimport os\nimport sys\nsys.path.append(os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__)))))\n\nfrom movery.analyzers.code_analyzer import CodeAnalyzer\n\nclass TestCodeAnalyzer(unittest.TestCase):\n def setUp(self):\n self.analyzer = CodeAnalyzer()\n self.test_data_dir = os.path.join(os.path.dirname(__file__), 'test_data')\n if not os.path.exists(self.test_data_dir):\n os.makedirs(self.test_data_dir)\n\n def test_parse_python(self):\n \"\"\"测试Python代码解析\"\"\"\n test_file = os.path.join(self.test_data_dir, 'test_python.py')\n with open(test_file, 'w') as f:\n f.write('''\n def example_function():\n x = 1\n y = 2\n return x + y\n ''')\n\n ast = self.analyzer.parse_file(test_file)\n self.assertIsNotNone(ast)\n self.assertEqual(ast.type, 'Module')\n\n def test_analyze_function(self):\n \"\"\"测试函数分析\"\"\"\n test_file = os.path.join(self.test_data_dir, 'test_function.py')\n with open(test_file, 'w') as f:\n f.write('''\n def process_data(data):\n result = []\n for item in data:\n if item > 0:\n result.append(item * 2)\n return result\n ''')\n\n functions = self.analyzer.analyze_functions(test_file)\n self.assertEqual(len(functions), 1)\n self.assertEqual(functions[0].name, 'process_data')\n self.assertTrue(functions[0].has_loop)\n self.assertTrue(functions[0].has_condition)\n\n def test_data_flow(self):\n \"\"\"测试数据流分析\"\"\"\n test_file = os.path.join(self.test_data_dir, 'test_dataflow.py')\n with open(test_file, 'w') as f:\n f.write('''\n def data_flow_example(user_input):\n data = user_input.strip()\n processed = data.lower()\n return processed\n ''')\n\n flows = self.analyzer.analyze_data_flow(test_file)\n self.assertTrue(len(flows) > 0)\n self.assertIn('user_input', flows[0].sources)\n self.assertIn('processed', flows[0].sinks)\n\n def test_complexity_analysis(self):\n \"\"\"测试复杂度分析\"\"\"\n test_file = os.path.join(self.test_data_dir, 'test_complexity.py')\n with open(test_file, 'w') as f:\n f.write('''\n def complex_function(x, y):\n if x > 0:\n if y > 0:\n return x + y\n else:\n return x - y\n else:\n if y < 0:\n return -x - y\n else:\n return -x + y\n ''')\n\n complexity = self.analyzer.analyze_complexity(test_file)\n self.assertTrue(complexity > 1)\n self.assertEqual(complexity, 4) # 4个条件分支\n\n def tearDown(self):\n \"\"\"清理测试数据\"\"\"\n import shutil\n if os.path.exists(self.test_data_dir):\n shutil.rmtree(self.test_data_dir)\n\nif __name__ == '__main__':\n unittest.main() " }, { "path": "movery/tests/unit/test_detector.py", "content": "import unittest\nimport os\nimport sys\nsys.path.append(os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__)))))\n\nfrom movery.detectors.vulnerability import VulnerabilityDetector\n\nclass TestVulnerabilityDetector(unittest.TestCase):\n def setUp(self):\n self.detector = VulnerabilityDetector()\n self.test_data_dir = os.path.join(os.path.dirname(__file__), 'test_data')\n if not os.path.exists(self.test_data_dir):\n os.makedirs(self.test_data_dir)\n\n def test_load_signatures(self):\n \"\"\"测试加载漏洞签名\"\"\"\n # 创建测试签名文件\n test_sig_file = os.path.join(self.test_data_dir, 'test_signatures.json')\n with open(test_sig_file, 'w') as f:\n f.write('''\n {\n \"signatures\": [\n {\n \"id\": \"CWE-78\",\n \"name\": \"OS Command Injection\",\n \"severity\": \"HIGH\",\n \"code_patterns\": [\"os\\\\.system\\\\(.*\\\\)\"]\n }\n ]\n }\n ''')\n \n self.detector.load_signatures(test_sig_file)\n self.assertEqual(len(self.detector.signatures), 1)\n self.assertEqual(self.detector.signatures[0].id, \"CWE-78\")\n\n def test_detect_vulnerability(self):\n \"\"\"测试漏洞检测\"\"\"\n # 创建测试代码文件\n test_code_file = os.path.join(self.test_data_dir, 'test_code.py')\n with open(test_code_file, 'w') as f:\n f.write('''\n import os\n def unsafe_function(cmd):\n os.system(cmd) # 不安全的系统命令执行\n ''')\n\n matches = self.detector.detect_file(test_code_file)\n self.assertTrue(len(matches) > 0)\n self.assertEqual(matches[0].signature.id, \"CWE-78\")\n\n def test_false_positive(self):\n \"\"\"测试误报情况\"\"\"\n # 创建安全的测试代码\n test_safe_file = os.path.join(self.test_data_dir, 'test_safe.py')\n with open(test_safe_file, 'w') as f:\n f.write('''\n def safe_function():\n print(\"This is safe code\")\n ''')\n\n matches = self.detector.detect_file(test_safe_file)\n self.assertEqual(len(matches), 0)\n\n def test_similarity_matching(self):\n \"\"\"测试相似度匹配\"\"\"\n # 创建相似代码测试文件\n test_similar_file = os.path.join(self.test_data_dir, 'test_similar.py')\n with open(test_similar_file, 'w') as f:\n f.write('''\n import subprocess\n def similar_unsafe(command):\n subprocess.call(command, shell=True) # 类似的不安全模式\n ''')\n\n matches = self.detector.detect_file(test_similar_file)\n self.assertTrue(len(matches) > 0)\n self.assertTrue(matches[0].confidence > 0.7)\n\n def tearDown(self):\n \"\"\"清理测试数据\"\"\"\n import shutil\n if os.path.exists(self.test_data_dir):\n shutil.rmtree(self.test_data_dir)\n\nif __name__ == '__main__':\n unittest.main() " }, { "path": "movery/tests/unit/test_security.py", "content": "import unittest\nimport os\nimport tempfile\nimport shutil\nimport time\nimport threading\nfrom movery.utils.security import SecurityChecker\n\nclass TestSecurityChecker(unittest.TestCase):\n def setUp(self):\n \"\"\"测试前的准备工作\"\"\"\n self.checker = SecurityChecker()\n self.test_dir = tempfile.mkdtemp()\n \n # 创建测试代码文件\n self.test_code = '''\nimport os\nimport sys\nimport time\nimport random\nimport socket\nimport subprocess\n\ndef unsafe_memory():\n # 大量内存分配\n large_list = [i for i in range(10**7)]\n return large_list\n\ndef unsafe_execution():\n # 长时间执行\n time.sleep(5)\n return \"Done\"\n\ndef unsafe_file_access():\n # 危险的文件操作\n with open(\"/etc/passwd\", \"r\") as f:\n data = f.read()\n return data\n\ndef unsafe_network():\n # 未经验证的网络连接\n sock = socket.socket()\n sock.connect((\"example.com\", 80))\n return sock\n\ndef unsafe_input():\n # 未验证的输入\n user_input = input(\"Enter command: \")\n os.system(user_input)\n\ndef unsafe_random():\n # 不安全的随机数生成\n return random.randint(1, 100)\n\ndef unsafe_sensitive_data():\n # 敏感数据暴露\n password = \"super_secret_123\"\n print(f\"Password is: {password}\")\n\ndef unsafe_sandbox():\n # 沙箱逃逸尝试\n subprocess.call(\"rm -rf /\", shell=True)\n'''\n self.test_file = os.path.join(self.test_dir, \"test_code.py\")\n with open(self.test_file, \"w\") as f:\n f.write(self.test_code)\n\n def tearDown(self):\n \"\"\"测试后的清理工作\"\"\"\n shutil.rmtree(self.test_dir)\n\n def test_check_memory_usage(self):\n \"\"\"测试内存使用检查\"\"\"\n result = self.checker.check_memory_usage(self.test_file)\n self.assertTrue(result[\"has_issues\"])\n self.assertIn(\"large_list\", result[\"details\"])\n self.assertGreater(len(result[\"patterns\"]), 0)\n\n def test_check_execution_time(self):\n \"\"\"测试执行时间检查\"\"\"\n result = self.checker.check_execution_time(self.test_file)\n self.assertTrue(result[\"has_issues\"])\n self.assertIn(\"time.sleep\", result[\"details\"])\n self.assertGreater(len(result[\"patterns\"]), 0)\n\n def test_check_file_access(self):\n \"\"\"测试文件访问检查\"\"\"\n result = self.checker.check_file_access(self.test_file)\n self.assertTrue(result[\"has_issues\"])\n self.assertIn(\"/etc/passwd\", result[\"details\"])\n self.assertGreater(len(result[\"patterns\"]), 0)\n\n def test_check_network_access(self):\n \"\"\"测试网络访问检查\"\"\"\n result = self.checker.check_network_access(self.test_file)\n self.assertTrue(result[\"has_issues\"])\n self.assertIn(\"socket.connect\", result[\"details\"])\n self.assertGreater(len(result[\"patterns\"]), 0)\n\n def test_check_input_validation(self):\n \"\"\"测试输入验证检查\"\"\"\n result = self.checker.check_input_validation(self.test_file)\n self.assertTrue(result[\"has_issues\"])\n self.assertIn(\"os.system\", result[\"details\"])\n self.assertGreater(len(result[\"patterns\"]), 0)\n\n def test_check_random_generation(self):\n \"\"\"测试随机数生成检查\"\"\"\n result = self.checker.check_random_generation(self.test_file)\n self.assertTrue(result[\"has_issues\"])\n self.assertIn(\"random.randint\", result[\"details\"])\n self.assertGreater(len(result[\"patterns\"]), 0)\n\n def test_check_sensitive_data(self):\n \"\"\"测试敏感数据检查\"\"\"\n result = self.checker.check_sensitive_data(self.test_file)\n self.assertTrue(result[\"has_issues\"])\n self.assertIn(\"password\", result[\"details\"])\n self.assertGreater(len(result[\"patterns\"]), 0)\n\n def test_check_sandbox_escape(self):\n \"\"\"测试沙箱逃逸检查\"\"\"\n result = self.checker.check_sandbox_escape(self.test_file)\n self.assertTrue(result[\"has_issues\"])\n self.assertIn(\"subprocess.call\", result[\"details\"])\n self.assertGreater(len(result[\"patterns\"]), 0)\n\n def test_perform_full_check(self):\n \"\"\"测试完整安全检查\"\"\"\n results = self.checker.perform_full_check(self.test_file)\n \n self.assertIsInstance(results, dict)\n self.assertGreater(len(results), 0)\n \n # 验证所有检查项都已执行\n expected_checks = [\n \"memory_usage\",\n \"execution_time\",\n \"file_access\",\n \"network_access\",\n \"input_validation\",\n \"random_generation\",\n \"sensitive_data\",\n \"sandbox_escape\"\n ]\n \n for check in expected_checks:\n self.assertIn(check, results)\n self.assertTrue(results[check][\"has_issues\"])\n self.assertGreater(len(results[check][\"patterns\"]), 0)\n\n def test_concurrent_checks(self):\n \"\"\"测试并发安全检查\"\"\"\n # 创建多个测试文件\n test_files = []\n for i in range(5):\n file_path = os.path.join(self.test_dir, f\"test_code_{i}.py\")\n with open(file_path, \"w\") as f:\n f.write(self.test_code)\n test_files.append(file_path)\n \n # 并发执行检查\n results = []\n threads = []\n \n def check_file(file_path):\n result = self.checker.perform_full_check(file_path)\n results.append(result)\n \n for file_path in test_files:\n thread = threading.Thread(target=check_file, args=(file_path,))\n threads.append(thread)\n thread.start()\n \n for thread in threads:\n thread.join()\n \n self.assertEqual(len(results), len(test_files))\n for result in results:\n self.assertIsInstance(result, dict)\n self.assertGreater(len(result), 0)\n\nif __name__ == '__main__':\n unittest.main() " }, { "path": "movery/tests/unit/test_vulnerability.py", "content": "import unittest\nimport os\nimport json\nimport tempfile\nimport shutil\nimport ast\nfrom movery.detectors.vulnerability import VulnerabilityDetector, Signature, VulnerabilityMatch\n\nclass TestVulnerabilityDetector(unittest.TestCase):\n def setUp(self):\n \"\"\"测试前的准备工作\"\"\"\n self.detector = VulnerabilityDetector()\n self.test_dir = tempfile.mkdtemp()\n \n # 创建测试签名文件\n self.signatures = {\n \"signatures\": [\n {\n \"id\": \"CMD001\",\n \"name\": \"命令注入\",\n \"severity\": \"high\",\n \"code_patterns\": [\n \"os\\\\.system\\\\([^)]*\\\\)\",\n \"subprocess\\\\.call\\\\([^)]*\\\\)\"\n ]\n },\n {\n \"id\": \"SQL001\",\n \"name\": \"SQL注入\",\n \"severity\": \"high\",\n \"code_patterns\": [\n \"execute\\\\(['\\\"][^'\\\"]*%[^'\\\"]*['\\\"]\\\\)\",\n \"executemany\\\\(['\\\"][^'\\\"]*%[^'\\\"]*['\\\"]\\\\)\"\n ]\n }\n ]\n }\n \n self.signature_file = os.path.join(self.test_dir, \"signatures.json\")\n with open(self.signature_file, \"w\") as f:\n json.dump(self.signatures, f)\n \n # 创建测试代码文件\n self.test_code = '''\nimport os\nimport subprocess\n\ndef unsafe_command():\n cmd = \"ls -l\"\n os.system(cmd)\n subprocess.call([\"echo\", \"hello\"])\n\ndef unsafe_sql():\n query = \"SELECT * FROM users WHERE id = %s\"\n cursor.execute(query % user_id)\n'''\n self.test_file = os.path.join(self.test_dir, \"test_code.py\")\n with open(self.test_file, \"w\") as f:\n f.write(self.test_code)\n\n def tearDown(self):\n \"\"\"测试后的清理工作\"\"\"\n shutil.rmtree(self.test_dir)\n\n def test_load_signatures(self):\n \"\"\"测试加载签名文件\"\"\"\n self.detector.load_signatures(self.signature_file)\n \n self.assertEqual(len(self.detector.signatures), 2)\n self.assertEqual(self.detector.signatures[0].id, \"CMD001\")\n self.assertEqual(self.detector.signatures[0].name, \"命令注入\")\n self.assertEqual(len(self.detector.signatures[0].code_patterns), 2)\n\n def test_detect_file(self):\n \"\"\"测试文件漏洞检测\"\"\"\n self.detector.load_signatures(self.signature_file)\n matches = self.detector.detect_file(self.test_file)\n \n self.assertGreater(len(matches), 0)\n for match in matches:\n self.assertIsInstance(match, VulnerabilityMatch)\n self.assertIsInstance(match.signature, Signature)\n self.assertGreater(match.confidence, 0.7)\n\n def test_analyze_ast(self):\n \"\"\"测试AST分析\"\"\"\n self.detector.load_signatures(self.signature_file)\n with open(self.test_file, 'r') as f:\n tree = ast.parse(f.read())\n matches = self.detector.analyze_ast(tree)\n \n self.assertGreater(len(matches), 0)\n for match in matches:\n self.assertIsInstance(match, VulnerabilityMatch)\n self.assertGreater(match.line_number, 0)\n\n def test_detect_similar_patterns(self):\n \"\"\"测试相似模式检测\"\"\"\n similar_code = '''\nimport os\nimport subprocess\n\ndef custom_system(cmd):\n os.system(cmd) # 直接模式\n \ndef modified_system(command):\n os.system(command) # 相似模式\n'''\n similar_file = os.path.join(self.test_dir, \"similar_code.py\")\n with open(similar_file, \"w\") as f:\n f.write(similar_code)\n \n self.detector.load_signatures(self.signature_file)\n matches = self.detector.detect_similar_patterns(similar_code)\n \n self.assertGreater(len(matches), 0)\n for match in matches:\n self.assertIsInstance(match, VulnerabilityMatch)\n self.assertGreater(match.confidence, 0.8)\n\n def test_calculate_confidence(self):\n \"\"\"测试置信度计算\"\"\"\n test_cases = [\n (\"os.system('ls')\", r\"os\\.system\\([^)]*\\)\", 0.8),\n (\"subprocess.call(['ls'])\", r\"subprocess\\.call\\([^)]*\\)\", 0.9),\n (\"import os; os.system('ls')\", r\"os\\.system\\([^)]*\\)\", 1.0)\n ]\n \n for code, pattern, expected in test_cases:\n confidence = self.detector._calculate_confidence(code, pattern)\n self.assertGreaterEqual(confidence, expected)\n self.assertLessEqual(confidence, 1.0)\n\n def test_calculate_similarity(self):\n \"\"\"测试相似度计算\"\"\"\n test_cases = [\n (\"os.system\", \"os.system\", 1.0),\n (\"os.system\", \"subprocess.system\", 0.5),\n (\"execute\", \"executemany\", 0.7)\n ]\n \n for str1, str2, expected in test_cases:\n similarity = self.detector._calculate_similarity(str1, str2)\n self.assertGreaterEqual(similarity, expected - 0.1)\n self.assertLessEqual(similarity, 1.0)\n\nif __name__ == '__main__':\n unittest.main() " }, { "path": "movery/utils/__init__.py", "content": "from .security import SecurityChecker\nfrom .parallel import WorkerPool, ParallelExecutor\nfrom .logging import get_logger\nfrom .memory import MemoryMonitor\n\n__all__ = ['SecurityChecker', 'WorkerPool', 'ParallelExecutor', 'get_logger', 'MemoryMonitor'] " }, { "path": "movery/utils/logging.py", "content": "\"\"\"\nLogging utilities for Movery\n\"\"\"\nimport logging\nimport sys\nimport os\nimport time\nfrom typing import Optional\nfrom datetime import datetime\nfrom functools import wraps\nimport threading\nfrom concurrent.futures import ThreadPoolExecutor\nimport queue\nimport json\n\nfrom movery.config.config import config\n\nclass AsyncLogHandler(logging.Handler):\n \"\"\"Asynchronous log handler that processes logs in a separate thread\"\"\"\n \n def __init__(self, capacity: int = 1000):\n super().__init__()\n self.queue = queue.Queue(maxsize=capacity)\n self.executor = ThreadPoolExecutor(max_workers=1)\n self.running = True\n self.worker = threading.Thread(target=self._process_logs)\n self.worker.daemon = True\n self.worker.start()\n \n def emit(self, record: logging.LogRecord):\n try:\n self.queue.put_nowait(record)\n except queue.Full:\n sys.stderr.write(f\"Log queue full, dropping message: {record.getMessage()}\\n\")\n \n def _process_logs(self):\n while self.running:\n try:\n record = self.queue.get(timeout=0.1)\n self.executor.submit(self._write_log, record)\n except queue.Empty:\n continue\n except Exception as e:\n sys.stderr.write(f\"Error processing log: {str(e)}\\n\")\n \n def _write_log(self, record: logging.LogRecord):\n try:\n message = self.format(record)\n with open(config.logging.log_file, \"a\", encoding=\"utf-8\") as f:\n f.write(message + \"\\n\")\n except Exception as e:\n sys.stderr.write(f\"Error writing log: {str(e)}\\n\")\n \n def close(self):\n self.running = False\n self.worker.join()\n self.executor.shutdown()\n super().close()\n\nclass ProgressLogger:\n \"\"\"Logger for tracking and displaying progress\"\"\"\n \n def __init__(self, total: int, desc: str = \"\", interval: float = 0.1):\n self.total = total\n self.desc = desc\n self.interval = interval\n self.current = 0\n self.start_time = time.time()\n self.last_update = 0\n \n def update(self, n: int = 1):\n self.current += n\n now = time.time()\n if now - self.last_update >= self.interval:\n self._display_progress()\n self.last_update = now\n \n def _display_progress(self):\n percentage = (self.current / self.total) * 100\n elapsed = time.time() - self.start_time\n rate = self.current / elapsed if elapsed > 0 else 0\n eta = (self.total - self.current) / rate if rate > 0 else 0\n \n sys.stdout.write(f\"\\r{self.desc}: [{self.current}/{self.total}] \"\n f\"{percentage:.1f}% Rate: {rate:.1f}/s ETA: {eta:.1f}s\")\n sys.stdout.flush()\n \n def finish(self):\n self._display_progress()\n sys.stdout.write(\"\\n\")\n sys.stdout.flush()\n\nclass JsonFormatter(logging.Formatter):\n \"\"\"Format logs as JSON for better parsing\"\"\"\n \n def format(self, record: logging.LogRecord) -> str:\n data = {\n \"timestamp\": datetime.fromtimestamp(record.created).isoformat(),\n \"level\": record.levelname,\n \"logger\": record.name,\n \"message\": record.getMessage(),\n \"module\": record.module,\n \"function\": record.funcName,\n \"line\": record.lineno\n }\n \n if record.exc_info:\n data[\"exception\"] = self.formatException(record.exc_info)\n \n if hasattr(record, \"extra\"):\n data.update(record.extra)\n \n return json.dumps(data)\n\ndef setup_logging(log_file: Optional[str] = None, level: str = None):\n \"\"\"Setup logging configuration\"\"\"\n if log_file:\n config.logging.log_file = log_file\n if level:\n config.logging.log_level = level\n \n # Create logs directory in current directory for relative paths\n if not os.path.isabs(config.logging.log_file):\n log_dir = os.path.join(os.getcwd(), \"logs\")\n config.logging.log_file = os.path.join(log_dir, config.logging.log_file)\n \n # Create log directory if needed\n os.makedirs(os.path.dirname(config.logging.log_file), exist_ok=True)\n \n # Setup root logger\n root_logger = logging.getLogger()\n root_logger.setLevel(config.logging.log_level)\n \n # Console handler\n console_handler = logging.StreamHandler(sys.stdout)\n console_handler.setLevel(logging.INFO)\n console_formatter = logging.Formatter(config.logging.log_format)\n console_handler.setFormatter(console_formatter)\n root_logger.addHandler(console_handler)\n \n # File handler\n file_handler = AsyncLogHandler()\n file_handler.setLevel(logging.DEBUG)\n file_formatter = JsonFormatter()\n file_handler.setFormatter(file_formatter)\n root_logger.addHandler(file_handler)\n \ndef log_execution_time(logger: Optional[logging.Logger] = None):\n \"\"\"Decorator to log function execution time\"\"\"\n def decorator(func):\n @wraps(func)\n def wrapper(*args, **kwargs):\n start_time = time.time()\n result = func(*args, **kwargs)\n elapsed_time = time.time() - start_time\n \n log = logger or logging.getLogger(func.__module__)\n log.debug(f\"{func.__name__} executed in {elapsed_time:.2f} seconds\")\n \n return result\n return wrapper\n return decorator\n\ndef get_logger(name: str) -> logging.Logger:\n \"\"\"Get a logger instance with the given name\"\"\"\n return logging.getLogger(name)\n\n# Initialize logging when module is imported\nsetup_logging() " }, { "path": "movery/utils/memory.py", "content": "\"\"\"\nMemory management utilities for Movery\n\"\"\"\nimport os\nimport mmap\nimport psutil\nimport gc\nimport sys\nfrom typing import Optional, Generator, Any\nfrom contextlib import contextmanager\nimport threading\nimport weakref\nfrom collections import OrderedDict\nimport logging\n\nfrom movery.config.config import config\n\nlogger = logging.getLogger(__name__)\n\nclass MemoryMonitor:\n \"\"\"Monitor memory usage and enforce limits\"\"\"\n \n def __init__(self, max_memory: Optional[int] = None):\n self.max_memory = max_memory or config.processing.max_memory_usage\n self.process = psutil.Process()\n self._lock = threading.Lock()\n self._last_check = 0\n \n def get_memory_usage(self) -> int:\n \"\"\"Get current memory usage in bytes\"\"\"\n return self.process.memory_info().rss\n \n def check_memory(self) -> bool:\n \"\"\"Check if memory usage is within limits\"\"\"\n with self._lock:\n current_usage = self.get_memory_usage()\n if current_usage > self.max_memory:\n logger.warning(f\"Memory usage ({current_usage} bytes) exceeds limit \"\n f\"({self.max_memory} bytes)\")\n return False\n return True\n \n def force_garbage_collection(self):\n \"\"\"Force garbage collection\"\"\"\n gc.collect()\n \n @contextmanager\n def monitor_operation(self, operation_name: str):\n \"\"\"Context manager to monitor memory during an operation\"\"\"\n start_usage = self.get_memory_usage()\n try:\n yield\n finally:\n end_usage = self.get_memory_usage()\n delta = end_usage - start_usage\n logger.debug(f\"Memory delta for {operation_name}: {delta} bytes\")\n if not self.check_memory():\n self.force_garbage_collection()\n\nclass LRUCache:\n \"\"\"Least Recently Used Cache with memory limit\"\"\"\n \n def __init__(self, max_size: Optional[int] = None):\n self.max_size = max_size or config.processing.cache_max_size\n self._cache = OrderedDict()\n self._size = 0\n self._lock = threading.Lock()\n \n def get(self, key: str) -> Optional[Any]:\n \"\"\"Get item from cache\"\"\"\n with self._lock:\n if key in self._cache:\n value = self._cache.pop(key)\n self._cache[key] = value\n return value\n return None\n \n def put(self, key: str, value: Any, size: Optional[int] = None):\n \"\"\"Put item in cache\"\"\"\n if not size:\n size = sys.getsizeof(value)\n \n if size > self.max_size:\n logger.warning(f\"Item size ({size} bytes) exceeds cache limit \"\n f\"({self.max_size} bytes)\")\n return\n \n with self._lock:\n if key in self._cache:\n self._size -= sys.getsizeof(self._cache[key])\n \n while self._size + size > self.max_size and self._cache:\n _, removed = self._cache.popitem(last=False)\n self._size -= sys.getsizeof(removed)\n \n self._cache[key] = value\n self._size += size\n \n def clear(self):\n \"\"\"Clear cache\"\"\"\n with self._lock:\n self._cache.clear()\n self._size = 0\n\nclass MemoryMappedFile:\n \"\"\"Memory mapped file for efficient large file handling\"\"\"\n \n def __init__(self, filename: str, mode: str = \"r\"):\n self.filename = filename\n self.mode = mode\n self._file = None\n self._mmap = None\n \n def __enter__(self):\n access = mmap.ACCESS_READ\n if \"w\" in self.mode:\n access = mmap.ACCESS_WRITE\n \n self._file = open(self.filename, mode=self.mode + \"b\")\n self._mmap = mmap.mmap(self._file.fileno(), 0, access=access)\n return self\n \n def __exit__(self, exc_type, exc_val, exc_tb):\n if self._mmap:\n self._mmap.close()\n if self._file:\n self._file.close()\n \n def read(self, size: int = -1) -> bytes:\n \"\"\"Read from memory mapped file\"\"\"\n if size == -1:\n return self._mmap[:]\n return self._mmap[:size]\n \n def write(self, data: bytes):\n \"\"\"Write to memory mapped file\"\"\"\n if \"w\" not in self.mode:\n raise IOError(\"File not opened for writing\")\n self._mmap.write(data)\n \n def seek(self, offset: int):\n \"\"\"Seek to position in file\"\"\"\n self._mmap.seek(offset)\n\ndef chunk_iterator(data: Any, chunk_size: Optional[int] = None) -> Generator:\n \"\"\"Iterator that yields chunks of data\"\"\"\n if not chunk_size:\n chunk_size = config.processing.chunk_size\n \n if isinstance(data, (bytes, str)):\n for i in range(0, len(data), chunk_size):\n yield data[i:i + chunk_size]\n elif hasattr(data, \"__iter__\"):\n chunk = []\n for item in data:\n chunk.append(item)\n if len(chunk) >= chunk_size:\n yield chunk\n chunk = []\n if chunk:\n yield chunk\n else:\n raise TypeError(f\"Unsupported data type: {type(data)}\")\n\n# Global memory monitor instance\nmemory_monitor = MemoryMonitor()\n\n# Global cache instance\ncache = LRUCache() " }, { "path": "movery/utils/parallel.py", "content": "\"\"\"\nParallel processing utilities for Movery\n\"\"\"\nimport multiprocessing as mp\nfrom multiprocessing import Pool, Queue, Manager\nfrom concurrent.futures import ProcessPoolExecutor, ThreadPoolExecutor\nimport threading\nfrom typing import Callable, List, Any, Optional, Dict, Tuple\nimport time\nimport os\nimport signal\nimport logging\nfrom functools import partial\nimport queue\nimport traceback\nfrom contextlib import contextmanager\n\nfrom movery.config.config import config\nfrom movery.utils.logging import get_logger\n\nlogger = get_logger(__name__)\n\nclass WorkerPool:\n \"\"\"Pool of worker processes with task queue\"\"\"\n \n def __init__(self, num_workers: Optional[int] = None,\n use_threads: bool = False):\n self.num_workers = num_workers or config.processing.num_processes\n self.use_threads = use_threads\n self._pool = None\n self._manager = None\n self._task_queue = None\n self._result_queue = None\n self._workers = []\n self._running = False\n self._lock = threading.Lock() if use_threads else mp.Lock()\n \n def start(self):\n \"\"\"Start worker pool\"\"\"\n if self._running:\n return\n \n self._manager = Manager() if not self.use_threads else None\n self._task_queue = Queue() if not self.use_threads else queue.Queue()\n self._result_queue = Queue() if not self.use_threads else queue.Queue()\n \n if self.use_threads:\n self._pool = ThreadPoolExecutor(max_workers=self.num_workers)\n else:\n self._pool = ProcessPoolExecutor(max_workers=self.num_workers)\n \n self._running = True\n logger.info(f\"Started worker pool with {self.num_workers} workers\")\n \n def stop(self):\n \"\"\"Stop worker pool\"\"\"\n if not self._running:\n return\n \n self._running = False\n if self._pool:\n self._pool.shutdown()\n if self._manager:\n self._manager.shutdown()\n \n logger.info(\"Stopped worker pool\")\n \n def submit(self, func: Callable, *args, **kwargs) -> Any:\n \"\"\"Submit task to worker pool\"\"\"\n if not self._running:\n raise RuntimeError(\"Worker pool not started\")\n \n future = self._pool.submit(func, *args, **kwargs)\n return future\n \n def map(self, func: Callable, iterable: List[Any]) -> List[Any]:\n \"\"\"Map function over iterable using worker pool\"\"\"\n if not self._running:\n raise RuntimeError(\"Worker pool not started\")\n \n return list(self._pool.map(func, iterable))\n \n def imap(self, func: Callable, iterable: List[Any]) -> Any:\n \"\"\"Iterator over mapped function results\"\"\"\n if not self._running:\n raise RuntimeError(\"Worker pool not started\")\n \n for result in self._pool.map(func, iterable):\n yield result\n \n @contextmanager\n def get_context(self):\n \"\"\"Context manager for worker pool\"\"\"\n self.start()\n try:\n yield self\n finally:\n self.stop()\n\nclass TaskQueue:\n \"\"\"Task queue with priority support\"\"\"\n \n def __init__(self, maxsize: int = 0):\n self.maxsize = maxsize\n self._queue = mp.Queue(maxsize=maxsize)\n self._unfinished_tasks = 0\n self._mutex = threading.Lock()\n self._not_empty = threading.Condition(self._mutex)\n self._not_full = threading.Condition(self._mutex)\n self._all_tasks_done = threading.Condition(self._mutex)\n \n def put(self, item: Any, priority: int = 0, block: bool = True,\n timeout: Optional[float] = None):\n \"\"\"Put item in queue with priority\"\"\"\n with self._not_full:\n if self.maxsize > 0:\n if not block:\n if self._qsize() >= self.maxsize:\n raise queue.Full\n elif timeout is None:\n while self._qsize() >= self.maxsize:\n self._not_full.wait()\n elif timeout < 0:\n raise ValueError(\"'timeout' must be a non-negative number\")\n else:\n endtime = time.time() + timeout\n while self._qsize() >= self.maxsize:\n remaining = endtime - time.time()\n if remaining <= 0.0:\n raise queue.Full\n self._not_full.wait(remaining)\n \n self._queue.put((priority, item))\n self._unfinished_tasks += 1\n self._not_empty.notify()\n \n def get(self, block: bool = True, timeout: Optional[float] = None) -> Any:\n \"\"\"Get item from queue\"\"\"\n with self._not_empty:\n if not block:\n if not self._qsize():\n raise queue.Empty\n elif timeout is None:\n while not self._qsize():\n self._not_empty.wait()\n elif timeout < 0:\n raise ValueError(\"'timeout' must be a non-negative number\")\n else:\n endtime = time.time() + timeout\n while not self._qsize():\n remaining = endtime - time.time()\n if remaining <= 0.0:\n raise queue.Empty\n self._not_empty.wait(remaining)\n \n item = self._queue.get()[1]\n self._not_full.notify()\n return item\n \n def task_done(self):\n \"\"\"Indicate that a task is done\"\"\"\n with self._all_tasks_done:\n unfinished = self._unfinished_tasks - 1\n if unfinished < 0:\n raise ValueError(\"task_done() called too many times\")\n self._unfinished_tasks = unfinished\n if unfinished == 0:\n self._all_tasks_done.notify_all()\n \n def join(self):\n \"\"\"Wait for all tasks to be done\"\"\"\n with self._all_tasks_done:\n while self._unfinished_tasks:\n self._all_tasks_done.wait()\n \n def qsize(self) -> int:\n \"\"\"Return queue size\"\"\"\n return self._queue.qsize()\n \n def empty(self) -> bool:\n \"\"\"Return True if queue is empty\"\"\"\n return self._queue.empty()\n \n def full(self) -> bool:\n \"\"\"Return True if queue is full\"\"\"\n return self._queue.full()\n \n def _qsize(self) -> int:\n \"\"\"Internal method to get queue size\"\"\"\n return self._queue.qsize()\n\nclass ParallelExecutor:\n \"\"\"Execute tasks in parallel with error handling\"\"\"\n \n def __init__(self, num_workers: Optional[int] = None,\n use_threads: bool = False):\n self.worker_pool = WorkerPool(num_workers, use_threads)\n self.task_queue = TaskQueue()\n self._results = {}\n self._errors = {}\n self._lock = threading.Lock()\n \n def submit(self, task_id: str, func: Callable, *args,\n priority: int = 0, **kwargs) -> None:\n \"\"\"Submit task for execution\"\"\"\n self.task_queue.put((task_id, func, args, kwargs), priority=priority)\n \n def execute(self) -> Tuple[Dict[str, Any], Dict[str, Exception]]:\n \"\"\"Execute all submitted tasks\"\"\"\n with self.worker_pool.get_context():\n while not self.task_queue.empty():\n try:\n task_id, func, args, kwargs = self.task_queue.get()\n future = self.worker_pool.submit(func, *args, **kwargs)\n future.add_done_callback(\n partial(self._handle_result, task_id))\n except Exception as e:\n logger.error(f\"Error executing task {task_id}: {str(e)}\")\n with self._lock:\n self._errors[task_id] = e\n finally:\n self.task_queue.task_done()\n \n self.task_queue.join()\n return self._results, self._errors\n \n def _handle_result(self, task_id: str, future):\n \"\"\"Handle task result or error\"\"\"\n try:\n result = future.result()\n with self._lock:\n self._results[task_id] = result\n except Exception as e:\n logger.error(f\"Error in task {task_id}: {str(e)}\")\n with self._lock:\n self._errors[task_id] = e\n\ndef parallel_map(func: Callable, iterable: List[Any],\n num_workers: Optional[int] = None,\n chunk_size: Optional[int] = None) -> List[Any]:\n \"\"\"Map function over iterable in parallel\"\"\"\n if not num_workers:\n num_workers = config.processing.num_processes\n if not chunk_size:\n chunk_size = max(1, len(iterable) // (num_workers * 4))\n \n with Pool(processes=num_workers) as pool:\n return pool.map(func, iterable, chunksize=chunk_size)\n\n# Global worker pool instance\nworker_pool = WorkerPool() " }, { "path": "movery/utils/security.py", "content": "import os\nimport ast\nimport time\nimport psutil\nimport socket\nimport threading\nfrom typing import List, Dict, Any, Optional\nimport re\nimport astroid\n\nclass SecurityChecker:\n \"\"\"安全检查器类\"\"\"\n \n def __init__(self):\n \"\"\"初始化安全检查器\"\"\"\n self.sensitive_patterns = {\n 'file_access': [\n r'open\\s*\\([^)]*[\\'\"]\\/(?:etc|root|home|usr|var)[^\\'\"]*[\\'\"]\\s*\\)',\n r'os\\.(?:remove|unlink|rmdir|mkdir|chmod|chown)',\n ],\n 'network_access': [\n r'socket\\.(?:socket|connect|bind|listen)',\n r'urllib\\.(?:request|urlopen)',\n r'requests\\.(?:get|post|put|delete)',\n ],\n 'code_execution': [\n r'(?:exec|eval|subprocess\\.(?:call|Popen|run))',\n r'os\\.(?:system|popen|spawn)',\n ],\n 'input_validation': [\n r'input\\s*\\(',\n r'raw_input\\s*\\(',\n r'eval\\s*\\(',\n ],\n 'random_generation': [\n r'random\\.(?:random|randint|choice|randrange)',\n r'secrets\\.(?:token_hex|token_bytes|token_urlsafe)',\n ],\n 'sensitive_data': [\n r'(?:password|secret|key|token|credential)',\n r'print\\s*\\([^)]*(?:password|secret|key|token)[^)]*\\)',\n ],\n }\n\n def check_memory_usage(self, file_path: str) -> Dict[str, Any]:\n \"\"\"检查文件执行时的内存使用情况\n\n Args:\n file_path: 待检查的文件路径\n\n Returns:\n Dict[str, Any]: 检查结果\n \"\"\"\n try:\n process = psutil.Process()\n initial_memory = process.memory_info().rss\n \n # 在新线程中执行代码以便监控\n def execute_code():\n with open(file_path, 'r') as f:\n exec(f.read())\n \n thread = threading.Thread(target=execute_code)\n thread.start()\n thread.join(timeout=5) # 最多等待5秒\n \n final_memory = process.memory_info().rss\n memory_usage = final_memory - initial_memory\n \n with open(file_path, 'r') as f:\n content = f.read()\n \n patterns = [r'list\\s*\\(.*\\)', r'\\[\\s*.*\\s*for\\s.*\\]', r'dict\\s*\\(.*\\)']\n matches = []\n for pattern in patterns:\n matches.extend(re.finditer(pattern, content))\n \n return {\n \"has_issues\": memory_usage > 100 * 1024 * 1024, # 超过100MB认为有问题\n \"issues\": [f\"内存使用量过大: {memory_usage / 1024 / 1024:.2f}MB\"] if memory_usage > 100 * 1024 * 1024 else [],\n \"details\": {match.group(): match.start() for match in matches},\n \"patterns\": patterns\n }\n except Exception as e:\n return {\n \"has_issues\": True,\n \"issues\": [f\"内存检查失败: {str(e)}\"],\n \"details\": {},\n \"patterns\": []\n }\n\n def check_execution_time(self, file_path: str, timeout: float = 5.0) -> Dict[str, Any]:\n \"\"\"检查文件执行时间\n\n Args:\n file_path: 待检查的文件路径\n timeout: 超时时间(秒)\n\n Returns:\n Dict[str, Any]: 检查结果\n \"\"\"\n try:\n start_time = time.time()\n \n def execute_code():\n with open(file_path, 'r') as f:\n exec(f.read())\n \n thread = threading.Thread(target=execute_code)\n thread.start()\n thread.join(timeout=timeout)\n \n execution_time = time.time() - start_time\n is_timeout = thread.is_alive()\n \n with open(file_path, 'r') as f:\n content = f.read()\n \n patterns = [r'while\\s+True:', r'time\\.sleep\\s*\\(', r'for\\s+.*\\s+in\\s+range\\s*\\(\\s*\\d+\\s*\\)']\n matches = []\n for pattern in patterns:\n matches.extend(re.finditer(pattern, content))\n \n return {\n \"has_issues\": is_timeout or execution_time > timeout,\n \"issues\": [f\"执行超时(>{timeout}秒)\"] if is_timeout else [f\"执行时间过长: {execution_time:.2f}秒\"] if execution_time > timeout else [],\n \"details\": {match.group(): match.start() for match in matches},\n \"patterns\": patterns\n }\n except Exception as e:\n return {\n \"has_issues\": True,\n \"issues\": [f\"执行时间检查失败: {str(e)}\"],\n \"details\": {},\n \"patterns\": []\n }\n\n def check_file_access(self, file_path: str) -> Dict[str, Any]:\n \"\"\"检查文件访问安全性\n\n Args:\n file_path: 待检查的文件路径\n\n Returns:\n Dict[str, Any]: 检查结果\n \"\"\"\n try:\n violations = []\n matches_dict = {}\n with open(file_path, 'r') as f:\n content = f.read()\n \n for pattern in self.sensitive_patterns['file_access']:\n matches = list(re.finditer(pattern, content))\n for match in matches:\n violations.append(f\"发现敏感文件操作: {match.group()}\")\n matches_dict[match.group()] = match.start()\n \n return {\n \"has_issues\": len(violations) > 0,\n \"issues\": violations,\n \"details\": matches_dict,\n \"patterns\": self.sensitive_patterns['file_access']\n }\n except Exception as e:\n return {\n \"has_issues\": True,\n \"issues\": [f\"文件访问检查失败: {str(e)}\"],\n \"details\": {},\n \"patterns\": []\n }\n\n def check_network_access(self, file_path: str) -> Dict[str, Any]:\n \"\"\"检查网络访问安全性\n\n Args:\n file_path: 待检查的文件路径\n\n Returns:\n Dict[str, Any]: 检查结果\n \"\"\"\n try:\n violations = []\n matches_dict = {}\n with open(file_path, 'r') as f:\n content = f.read()\n \n for pattern in self.sensitive_patterns['network_access']:\n matches = list(re.finditer(pattern, content))\n for match in matches:\n violations.append(f\"发现敏感网络操作: {match.group()}\")\n matches_dict[match.group()] = match.start()\n \n return {\n \"has_issues\": len(violations) > 0,\n \"issues\": violations,\n \"details\": matches_dict,\n \"patterns\": self.sensitive_patterns['network_access']\n }\n except Exception as e:\n return {\n \"has_issues\": True,\n \"issues\": [f\"网络访问检查失败: {str(e)}\"],\n \"details\": {},\n \"patterns\": []\n }\n\n def check_input_validation(self, file_path: str) -> Dict[str, Any]:\n \"\"\"检查输入验证\n\n Args:\n file_path: 待检查的文件路径\n\n Returns:\n Dict[str, Any]: 检查结果\n \"\"\"\n try:\n issues = []\n matches_dict = {}\n with open(file_path, 'r') as f:\n content = f.read()\n \n try:\n module = ast.parse(content)\n for node in ast.walk(module):\n if isinstance(node, ast.Call):\n if isinstance(node.func, ast.Name):\n func_name = node.func.id\n if func_name in ['input', 'raw_input']:\n issues.append(f\"未验证的输入: 第{node.lineno}行\")\n matches_dict[func_name] = node.lineno\n elif isinstance(node.func, ast.Attribute):\n if node.func.attr in ['get', 'post', 'put', 'delete']:\n issues.append(f\"未验证的HTTP请求: 第{node.lineno}行\")\n matches_dict[f\"{node.func.value.id}.{node.func.attr}\"] = node.lineno\n except:\n issues.append(\"代码解析失败\")\n \n return {\n \"has_issues\": len(issues) > 0,\n \"issues\": issues,\n \"details\": matches_dict,\n \"patterns\": self.sensitive_patterns['input_validation']\n }\n except Exception as e:\n return {\n \"has_issues\": True,\n \"issues\": [f\"输入验证检查失败: {str(e)}\"],\n \"details\": {},\n \"patterns\": []\n }\n\n def check_random_generation(self, file_path: str) -> Dict[str, Any]:\n \"\"\"检查随机数生成安全性\n\n Args:\n file_path: 待检查的文件路径\n\n Returns:\n Dict[str, Any]: 检查结果\n \"\"\"\n try:\n issues = []\n matches_dict = {}\n with open(file_path, 'r') as f:\n content = f.read()\n \n for pattern in self.sensitive_patterns['random_generation']:\n matches = list(re.finditer(pattern, content))\n for match in matches:\n if 'secrets' not in match.group():\n issues.append(f\"不安全的随机数生成: {match.group()}\")\n matches_dict[match.group()] = match.start()\n \n return {\n \"has_issues\": len(issues) > 0,\n \"issues\": issues,\n \"details\": matches_dict,\n \"patterns\": self.sensitive_patterns['random_generation']\n }\n except Exception as e:\n return {\n \"has_issues\": True,\n \"issues\": [f\"随机数生成检查失败: {str(e)}\"],\n \"details\": {},\n \"patterns\": []\n }\n\n def check_sensitive_data(self, file_path: str) -> Dict[str, Any]:\n \"\"\"检查敏感数据处理\n\n Args:\n file_path: 待检查的文件路径\n\n Returns:\n Dict[str, Any]: 检查结果\n \"\"\"\n try:\n issues = []\n matches_dict = {}\n with open(file_path, 'r') as f:\n content = f.read()\n \n for pattern in self.sensitive_patterns['sensitive_data']:\n matches = list(re.finditer(pattern, content))\n for match in matches:\n issues.append(f\"敏感数据泄露风险: {match.group()}\")\n matches_dict[match.group()] = match.start()\n \n return {\n \"has_issues\": len(issues) > 0,\n \"issues\": issues,\n \"details\": matches_dict,\n \"patterns\": self.sensitive_patterns['sensitive_data']\n }\n except Exception as e:\n return {\n \"has_issues\": True,\n \"issues\": [f\"敏感数据检查失败: {str(e)}\"],\n \"details\": {},\n \"patterns\": []\n }\n\n def check_sandbox_escape(self, file_path: str) -> Dict[str, Any]:\n \"\"\"检查沙箱逃逸\n\n Args:\n file_path: 待检查的文件路径\n\n Returns:\n Dict[str, Any]: 检查结果\n \"\"\"\n try:\n violations = []\n matches_dict = {}\n with open(file_path, 'r') as f:\n content = f.read()\n \n try:\n module = astroid.parse(content)\n for node in module.nodes_of_class(astroid.Call):\n if isinstance(node.func, astroid.Attribute):\n if node.func.attrname in ['system', 'popen', 'spawn', 'call', 'Popen', 'run']:\n violations.append(f\"危险的系统调用: {node.as_string()}\")\n matches_dict[node.as_string()] = node.lineno\n elif isinstance(node.func, astroid.Name):\n if node.func.name in ['exec', 'eval']:\n violations.append(f\"危险的代码执行: {node.as_string()}\")\n matches_dict[node.as_string()] = node.lineno\n except:\n violations.append(\"代码解析失败\")\n \n return {\n \"has_issues\": len(violations) > 0,\n \"issues\": violations,\n \"details\": matches_dict,\n \"patterns\": self.sensitive_patterns['code_execution']\n }\n except Exception as e:\n return {\n \"has_issues\": True,\n \"issues\": [f\"沙箱逃逸检查失败: {str(e)}\"],\n \"details\": {},\n \"patterns\": []\n }\n\n def perform_full_check(self, file_path: str) -> Dict[str, Any]:\n \"\"\"执行完整的安全检查\n\n Args:\n file_path: 待检查的文件路径\n\n Returns:\n Dict[str, Any]: 检查结果\n \"\"\"\n results = {\n 'memory_usage': self.check_memory_usage(file_path),\n 'execution_time': self.check_execution_time(file_path),\n 'file_access': self.check_file_access(file_path),\n 'network_access': self.check_network_access(file_path),\n 'input_validation': self.check_input_validation(file_path),\n 'random_generation': self.check_random_generation(file_path),\n 'sensitive_data': self.check_sensitive_data(file_path),\n 'sandbox_escape': self.check_sandbox_escape(file_path)\n }\n \n return results\n" }, { "path": "requirements.txt", "content": "jinja2>=3.0.0\nplotly>=5.0.0\npandas>=1.3.0\npsutil>=5.8.0\ntqdm>=4.61.0\ncolorama>=0.4.4\nrequests>=2.26.0\nbeautifulsoup4>=4.9.3\nlxml>=4.6.3\npygments>=2.9.0\ntyping-extensions>=3.10.0\ndataclasses>=0.8;python_version<\"3.7\"\nastroid>=2.15.0\n" }, { "path": "setup.py", "content": "\"\"\"\nSetup script for Re-Movery\n\"\"\"\nfrom setuptools import setup, find_packages\n\nwith open(\"README.md\", \"r\", encoding=\"utf-8\") as f:\n long_description = f.read()\n\nsetup(\n name=\"movery\",\n version=\"0.1.0\",\n author=\"heyangxu\",\n author_email=\"\",\n description=\"A tool for discovering modified vulnerable code clones\",\n long_description=long_description,\n long_description_content_type=\"text/markdown\",\n url=\"https://github.com/heyangxu/Re-movery\",\n packages=find_packages(),\n classifiers=[\n \"Development Status :: 4 - Beta\",\n \"Intended Audience :: Developers\",\n \"Topic :: Security\",\n \"Topic :: Software Development :: Quality Assurance\",\n \"License :: OSI Approved :: MIT License\",\n \"Programming Language :: Python :: 3\",\n \"Programming Language :: Python :: 3.7\",\n \"Programming Language :: Python :: 3.8\",\n \"Programming Language :: Python :: 3.9\",\n \"Programming Language :: Python :: 3.10\",\n \"Operating System :: OS Independent\",\n ],\n python_requires=\">=3.7\",\n install_requires=[\n \"pytest>=7.3.1\",\n \"coverage>=7.2.7\",\n \"jinja2>=3.0.0\",\n \"plotly>=5.0.0\",\n \"pandas>=1.3.0\",\n \"psutil>=5.8.0\",\n \"tqdm>=4.61.0\",\n \"colorama>=0.4.4\",\n \"requests>=2.26.0\",\n \"beautifulsoup4>=4.9.3\",\n \"lxml>=4.6.3\",\n \"pygments>=2.9.0\",\n \"typing-extensions>=3.10.0\",\n \"dataclasses>=0.8;python_version<'3.7'\",\n ],\n entry_points={\n \"console_scripts\": [\n \"movery=movery.main:main\",\n ],\n },\n package_data={\n \"movery\": [\n \"templates/*.html\",\n \"config/*.json\",\n ],\n },\n include_package_data=True,\n zip_safe=False,\n) " }, { "path": "signatures.json", "content": "{\n \"signatures\": [\n {\n \"id\": \"CWE-78\",\n \"name\": \"OS Command Injection\",\n \"description\": \"The software constructs all or part of an OS command using externally-influenced input from an upstream component, but it does not neutralize or incorrectly neutralizes special elements that could modify the intended OS command when it is sent to a downstream component.\",\n \"severity\": \"CRITICAL\",\n \"cwe_id\": \"CWE-78\",\n \"affected_languages\": [\"python\", \"php\", \"javascript\"],\n \"code_patterns\": [\n \"os\\\\.system\\\\(.*\\\\)\",\n \"subprocess\\\\.call\\\\(.*shell\\\\s*=\\\\s*True.*\\\\)\",\n \"exec\\\\(.*\\\\)\",\n \"eval\\\\(.*\\\\)\"\n ],\n \"fix_patterns\": [\n \"shlex.quote(command)\",\n \"subprocess.run([command], shell=False)\",\n \"ast.literal_eval(input)\"\n ],\n \"context_patterns\": [\n \"import\\\\s+os\",\n \"import\\\\s+subprocess\",\n \"import\\\\s+shlex\"\n ]\n },\n {\n \"id\": \"CWE-89\",\n \"name\": \"SQL Injection\",\n \"description\": \"The software constructs all or part of an SQL command using externally-influenced input from an upstream component, but it does not neutralize or incorrectly neutralizes special elements that could modify the intended SQL command when it is sent to a downstream component.\",\n \"severity\": \"CRITICAL\",\n \"cwe_id\": \"CWE-89\",\n \"affected_languages\": [\"python\", \"php\", \"java\"],\n \"code_patterns\": [\n \"cursor\\\\.execute\\\\(.*%.*\\\\)\",\n \"cursor\\\\.execute\\\\(.*\\\\+.*\\\\)\",\n \"cursor\\\\.executemany\\\\(.*%.*\\\\)\",\n \"mysql_query\\\\(.*\\\\$.*\\\\)\"\n ],\n \"fix_patterns\": [\n \"cursor.execute(query, params)\",\n \"cursor.executemany(query, params)\",\n \"prepared_statement.setString(1, input)\"\n ],\n \"context_patterns\": [\n \"import\\\\s+sqlite3\",\n \"import\\\\s+mysql\",\n \"import\\\\s+psycopg2\"\n ]\n },\n {\n \"id\": \"CWE-22\",\n \"name\": \"Path Traversal\",\n \"description\": \"The software uses external input to construct a pathname that is intended to identify a file or directory that is located underneath a restricted parent directory, but the software does not properly neutralize special elements within the pathname that can cause the pathname to resolve to a location that is outside of the restricted directory.\",\n \"severity\": \"HIGH\",\n \"cwe_id\": \"CWE-22\",\n \"affected_languages\": [\"python\", \"php\", \"java\", \"javascript\"],\n \"code_patterns\": [\n \"open\\\\(.*\\\\+.*\\\\)\",\n \"file_get_contents\\\\(.*\\\\$.*\\\\)\",\n \"new\\\\s+File\\\\(.*\\\\+.*\\\\)\"\n ],\n \"fix_patterns\": [\n \"os.path.abspath(os.path.join(base_dir, filename))\",\n \"os.path.normpath(path)\",\n \"Path(path).resolve().is_relative_to(base_dir)\"\n ],\n \"context_patterns\": [\n \"import\\\\s+os\",\n \"from\\\\s+pathlib\\\\s+import\\\\s+Path\"\n ]\n },\n {\n \"id\": \"CWE-79\",\n \"name\": \"Cross-site Scripting (XSS)\",\n \"description\": \"The software does not neutralize or incorrectly neutralizes user-controllable input before it is placed in output that is used as a web page that is served to other users.\",\n \"severity\": \"HIGH\",\n \"cwe_id\": \"CWE-79\",\n \"affected_languages\": [\"python\", \"php\", \"javascript\"],\n \"code_patterns\": [\n \"innerHTML\\\\s*=.*\",\n \"document\\\\.write\\\\(.*\\\\)\",\n \"\\\\$\\\\(.*\\\\)\\\\.html\\\\(.*\\\\)\"\n ],\n \"fix_patterns\": [\n \"textContent = content\",\n \"innerText = content\",\n \"createElement('div')\"\n ],\n \"context_patterns\": [\n \"