Repository: nashsu/FreeAskInternet Branch: main Commit: 3fa58d17e908 Files: 10 Total size: 95.7 KB Directory structure: gitextract_mcwgdhv6/ ├── .gitignore ├── Dockerfile ├── LICENSE ├── README.md ├── docker-compose.yaml ├── free_ask_internet.py ├── requirements.txt ├── searxng/ │ ├── settings.yml │ └── uwsgi.ini └── server.py ================================================ FILE CONTENTS ================================================ ================================================ FILE: .gitignore ================================================ __pycache__/ *.py[cod] *$py.class ================================================ FILE: Dockerfile ================================================ FROM python:3.9.15 WORKDIR /app COPY requirements.txt /app RUN pip3 install -i https://mirrors.aliyun.com/pypi/simple/ --trusted-host mirrors.aliyun.com -r requirements.txt --no-cache-dir COPY . /app EXPOSE 8000 ENTRYPOINT ["python3"] CMD ["server.py"] ================================================ FILE: LICENSE ================================================ Apache License Version 2.0, January 2004 http://www.apache.org/licenses/ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 1. Definitions. "License" shall mean the terms and conditions for use, reproduction, and distribution as defined by Sections 1 through 9 of this document. "Licensor" shall mean the copyright owner or entity authorized by the copyright owner that is granting the License. "Legal Entity" shall mean the union of the acting entity and all other entities that control, are controlled by, or are under common control with that entity. For the purposes of this definition, "control" means (i) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the outstanding shares, or (iii) beneficial ownership of such entity. "You" (or "Your") shall mean an individual or Legal Entity exercising permissions granted by this License. "Source" form shall mean the preferred form for making modifications, including but not limited to software source code, documentation source, and configuration files. "Object" form shall mean any form resulting from mechanical transformation or translation of a Source form, including but not limited to compiled object code, generated documentation, and conversions to other media types. "Work" shall mean the work of authorship, whether in Source or Object form, made available under the License, as indicated by a copyright notice that is included in or attached to the work (an example is provided in the Appendix below). "Derivative Works" shall mean any work, whether in Source or Object form, that is based on (or derived from) the Work and for which the editorial revisions, annotations, elaborations, or other modifications represent, as a whole, an original work of authorship. For the purposes of this License, Derivative Works shall not include works that remain separable from, or merely link (or bind by name) to the interfaces of, the Work and Derivative Works thereof. "Contribution" shall mean any work of authorship, including the original version of the Work and any modifications or additions to that Work or Derivative Works thereof, that is intentionally submitted to Licensor for inclusion in the Work by the copyright owner or by an individual or Legal Entity authorized to submit on behalf of the copyright owner. For the purposes of this definition, "submitted" means any form of electronic, verbal, or written communication sent to the Licensor or its representatives, including but not limited to communication on electronic mailing lists, source code control systems, and issue tracking systems that are managed by, or on behalf of, the Licensor for the purpose of discussing and improving the Work, but excluding communication that is conspicuously marked or otherwise designated in writing by the copyright owner as "Not a Contribution." "Contributor" shall mean Licensor and any individual or Legal Entity on behalf of whom a Contribution has been received by Licensor and subsequently incorporated within the Work. 2. Grant of Copyright License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare Derivative Works of, publicly display, publicly perform, sublicense, and distribute the Work and such Derivative Works in Source or Object form. 3. Grant of Patent License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Work, where such license applies only to those patent claims licensable by such Contributor that are necessarily infringed by their Contribution(s) alone or by combination of their Contribution(s) with the Work to which such Contribution(s) was submitted. If You institute patent litigation against any entity (including a cross-claim or counterclaim in a lawsuit) alleging that the Work or a Contribution incorporated within the Work constitutes direct or contributory patent infringement, then any patent licenses granted to You under this License for that Work shall terminate as of the date such litigation is filed. 4. Redistribution. You may reproduce and distribute copies of the Work or Derivative Works thereof in any medium, with or without modifications, and in Source or Object form, provided that You meet the following conditions: (a) You must give any other recipients of the Work or Derivative Works a copy of this License; and (b) You must cause any modified files to carry prominent notices stating that You changed the files; and (c) You must retain, in the Source form of any Derivative Works that You distribute, all copyright, patent, trademark, and attribution notices from the Source form of the Work, excluding those notices that do not pertain to any part of the Derivative Works; and (d) If the Work includes a "NOTICE" text file as part of its distribution, then any Derivative Works that You distribute must include a readable copy of the attribution notices contained within such NOTICE file, excluding those notices that do not pertain to any part of the Derivative Works, in at least one of the following places: within a NOTICE text file distributed as part of the Derivative Works; within the Source form or documentation, if provided along with the Derivative Works; or, within a display generated by the Derivative Works, if and wherever such third-party notices normally appear. The contents of the NOTICE file are for informational purposes only and do not modify the License. You may add Your own attribution notices within Derivative Works that You distribute, alongside or as an addendum to the NOTICE text from the Work, provided that such additional attribution notices cannot be construed as modifying the License. You may add Your own copyright statement to Your modifications and may provide additional or different license terms and conditions for use, reproduction, or distribution of Your modifications, or for any such Derivative Works as a whole, provided Your use, reproduction, and distribution of the Work otherwise complies with the conditions stated in this License. 5. Submission of Contributions. Unless You explicitly state otherwise, any Contribution intentionally submitted for inclusion in the Work by You to the Licensor shall be under the terms and conditions of this License, without any additional terms or conditions. Notwithstanding the above, nothing herein shall supersede or modify the terms of any separate license agreement you may have executed with Licensor regarding such Contributions. 6. Trademarks. This License does not grant permission to use the trade names, trademarks, service marks, or product names of the Licensor, except as required for reasonable and customary use in describing the origin of the Work and reproducing the content of the NOTICE file. 7. Disclaimer of Warranty. Unless required by applicable law or agreed to in writing, Licensor provides the Work (and each Contributor provides its Contributions) on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, including, without limitation, any warranties or conditions of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE. You are solely responsible for determining the appropriateness of using or redistributing the Work and assume any risks associated with Your exercise of permissions under this License. 8. Limitation of Liability. In no event and under no legal theory, whether in tort (including negligence), contract, or otherwise, unless required by applicable law (such as deliberate and grossly negligent acts) or agreed to in writing, shall any Contributor be liable to You for damages, including any direct, indirect, special, incidental, or consequential damages of any character arising as a result of this License or out of the use or inability to use the Work (including but not limited to damages for loss of goodwill, work stoppage, computer failure or malfunction, or any and all other commercial damages or losses), even if such Contributor has been advised of the possibility of such damages. 9. Accepting Warranty or Additional Liability. While redistributing the Work or Derivative Works thereof, You may choose to offer, and charge a fee for, acceptance of support, warranty, indemnity, or other liability obligations and/or rights consistent with this License. However, in accepting such obligations, You may act only on Your own behalf and on Your sole responsibility, not on behalf of any other Contributor, and only if You agree to indemnify, defend, and hold each Contributor harmless for any liability incurred by, or claims asserted against, such Contributor by reason of your accepting any such warranty or additional liability. END OF TERMS AND CONDITIONS APPENDIX: How to apply the Apache License to your work. To apply the Apache License to your work, attach the following boilerplate notice, with the fields enclosed by brackets "[]" replaced with your own identifying information. (Don't include the brackets!) The text should be enclosed in the appropriate comment syntax for the file format. We also recommend that a file or class name and description of purpose be included on the same "printed page" as the copyright notice for easier identification within third-party archives. Copyright [yyyy] [name of copyright owner] Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. ================================================ FILE: README.md ================================================ # FreeAskInternet ## 🎉🎉🎉 Yeah we have a logo now! 🎉🎉🎉 ![lgoo](./doc/logo-20240412.png) > Running www.perplexity.ai like app complete FREE, LOCAL, PRIVATE and NO GPU NEED on any computer > [!IMPORTANT] > **If you are unable to use this project normally, it is most likely due to issues with your internet connection or your IP, you need free internet connection to use this project normally. 如果您无法正常使用此项目，很可能是由于您的 IP 存在问题，或者你不能自由访问互联网。** ## What is FreeAskInternet FreeAskInternet is a completely free, private and locally running search aggregator & answer generate using LLM, Without GPU needed. The user can ask a question and the system will use searxng to make a multi engine search and combine the search result to the ChatGPT3.5 LLM and generate the answer based on search results. All process running locally and No GPU or OpenAI or Google API keys are needed. ## Features - 🈚️ Completely FREE (no need for any API keys) - 💻 Completely LOCAL (no GPU need, any computer can run ) - 🔐 Completely PRIVATE (all thing running locally, using custom llm) - 👻 Runs WITHOUT LLM Hardware (NO GPU NEED!) - 🤩 Using Free ChatGPT3.5 / Qwen / Kimi / ZhipuAI(GLM) API (NO API keys need! Thx OpenAI) - 🐵 Custom LLM(ollama,llama.cpp) support, Yes we love ollama! - 🚀 Fast and easy to deploy with Docker Compose - 🌐 Web and Mobile friendly interface, designed for Web Search enhanced AI Chat, allowing for easy access from any device. ## Screenshots 1. index: ![index](./doc/index.png) 2. Search based AI Chat: ![index](./doc/Screenshot-4.png) 3. Multi LLM models and custom LLM like ollama support: ![index](./doc/Screenshot-5.png) ## How It Works? 1. System get user input question in FreeAskInternet UI interface( running locally), and call searxng (running locally) to make search on multi search engine. 2. crawl search result links content and pass to ChatGPT3.5 / Kimi / Qwen / ZhipuAI / ollama (by using custom llm), ask LLM to answer user question based on this contents as references. 3. Stream the answer to Chat UI. 4. We support custom LLM setting, so theoretically infinite llm support. ## Status This project is still in its very early days. Expect some bugs. ### Run the latest release ```bash git clone https://github.com/nashsu/FreeAskInternet.git cd ./FreeAskInternet docker-compose up -d ``` 🎉 You should now be able to open the web interface on http://localhost:3000. Nothing else is exposed by default.( For old web interface, accessing http://localhost:3030) ## How to get and set Kimi / Qwen / ZhipuAI Token? How to get Token? We are using [https://github.com/LLM-Red-Team](https://github.com/LLM-Red-Team) projects to provide those service, you can reference to their readme. Reference : [https://github.com/LLM-Red-Team/kimi-free-api](https://github.com/LLM-Red-Team/kimi-free-api) ![setting token](./doc/Screenshot-3.png) ## How to using custom LLM like ollama? (Yes we love ollama) 1. start ollama serve ```bash export OLLAMA_HOST=0.0.0.0 ollama serve ``` 2. set ollama url in setting: You MUST using your computer's ip address, not localhost/127.0.0.1, because in docker you can't access this address. The model name is the model you want to serve by ollama. ![setting custom llm url](./doc/Screenshot-2.png) ollama model Reference : [https://ollama.com/library](https://ollama.com/library) ### How to update to latest ```bash cd ./FreeAskInternet git pull docker compose down docker compose rm backend docker compose rm free_ask_internet_ui docker image rm nashsu/free_ask_internet docker image rm nashsu/free_ask_internet_ui docker-compose up -d ``` ## Credits - ChatGPT-Next-Web : [https://github.com/ChatGPTNextWeb/ChatGPT-Next-Web](https://github.com/ChatGPTNextWeb/ChatGPT-Next-Web) - FreeGPT35: [https://github.com/missuo/FreeGPT35](https://github.com/missuo/FreeGPT35) - Kimi\Qwen\ZhipuAI [https://github.com/LLM-Red-Team](https://github.com/LLM-Red-Team) - searxng: [https://github.com/searxng/searxng](https://github.com/searxng/searxng) ## Special thanks to our logo designer [AdlerMurcus](https://github.com/AdlerMurcus) ## License Apache-2.0 license ## Star History [![Star History Chart](https://api.star-history.com/svg?repos=nashsu/FreeAskInternet&type=Date)](https://star-history.com/#nashsu/FreeAskInternet&Date) ================================================ FILE: docker-compose.yaml ================================================ services: backend: image: docker.io/nashsu/free_ask_internet:latest depends_on: - llm-freegpt35 restart: on-failure freeaskinternet-ui: image: docker.io/nashsu/free_ask_internet_ui:latest ports: - "3000:80" environment: BACKEND_HOST: "backend:8000" depends_on: - backend restart: always chatgpt-next-web: image: yidadaa/chatgpt-next-web ports: - "3030:3000" environment: OPENAI_API_KEY: "FreeAskInternet" # CODE: "FreeAskInternet" # 如果你想要设置页面的访问密码，请修改这里 BASE_URL: "http://backend:8000" CUSTOM_MODELS: "-all,+gpt-3.5-turbo" depends_on: - llm-freegpt35 restart: always llm-freegpt35: image: missuo/freegpt35:latest restart: always llm-kimi: image: vinlic/kimi-free-api:latest restart: always environment: - TZ=Asia/Shanghai llm-glm4: image: vinlic/glm-free-api:latest restart: always environment: - TZ=Asia/Shanghai llm-qwen: image: vinlic/qwen-free-api:latest restart: always environment: - TZ=Asia/Shanghai searxng: image: docker.io/searxng/searxng:latest volumes: - ./searxng:/etc/searxng:rw environment: - SEARXNG_BASE_URL=https://${SEARXNG_HOSTNAME:-localhost}/ cap_drop: - ALL cap_add: - CHOWN - SETGID - SETUID logging: driver: 'json-file' options: max-size: '1m' max-file: '1' restart: always ================================================ FILE: free_ask_internet.py ================================================ # -*- coding: utf-8 -*- import json import os from pprint import pprint import requests import trafilatura from trafilatura import bare_extraction from concurrent.futures import ThreadPoolExecutor import concurrent import requests import openai import time from datetime import datetime from urllib.parse import urlparse import tldextract import platform import urllib.parse def extract_url_content(url): downloaded = trafilatura.fetch_url(url) content = trafilatura.extract(downloaded) return {"url":url, "content":content} def search_web_ref(query:str, debug=False): content_list = [] try: safe_string = urllib.parse.quote_plus(":all !general " + query) response = requests.get('http://searxng:8080?q=' + safe_string + '&format=json') response.raise_for_status() search_results = response.json() if debug: print("JSON Response:") pprint(search_results) pedding_urls = [] conv_links = [] if search_results.get('results'): for item in search_results.get('results')[0:9]: name = item.get('title') snippet = item.get('content') url = item.get('url') pedding_urls.append(url) if url: url_parsed = urlparse(url) domain = url_parsed.netloc icon_url = url_parsed.scheme + '://' + url_parsed.netloc + '/favicon.ico' site_name = tldextract.extract(url).domain conv_links.append({ 'site_name':site_name, 'icon_url':icon_url, 'title':name, 'url':url, 'snippet':snippet }) results = [] futures = [] executor = ThreadPoolExecutor(max_workers=10) for url in pedding_urls: futures.append(executor.submit(extract_url_content,url)) try: for future in futures: res = future.result(timeout=5) results.append(res) except concurrent.futures.TimeoutError: print("任务执行超时") executor.shutdown(wait=False,cancel_futures=True) for content in results: if content and content.get('content'): item_dict = { "url":content.get('url'), "content": content.get('content'), "length":len(content.get('content')) } content_list.append(item_dict) if debug: print("URL: {}".format(url)) print("=================") return conv_links,content_list except Exception as ex: raise ex def gen_prompt(question,content_list, lang="zh-CN", context_length_limit=11000,debug=False): limit_len = (context_length_limit - 2000) if len(question) > limit_len: question = question[0:limit_len] ref_content = [ item.get("content") for item in content_list] answer_language = ' Simplified Chinese ' if lang == "zh-CN": answer_language = ' Simplified Chinese ' if lang == "zh-TW": answer_language = ' Traditional Chinese ' if lang == "en-US": answer_language = ' English ' if len(ref_content) > 0: if False: prompts = ''' 您是一位由 nash_su 开发的大型语言人工智能助手。您将被提供一个用户问题，并需要撰写一个清晰、简洁且准确的答案。提供了一组与问题相关的上下文，每个都以[[citation:x]]这样的编号开头，x代表一个数字。请在适当的情况下在句子末尾引用上下文。答案必须正确、精确，并以专家的中立和职业语气撰写。请将答案限制在2000个标记内。不要提供与问题无关的信息，也不要重复。如果给出的上下文信息不足，请在相关主题后写上“信息缺失：”。请按照引用编号[citation:x]的格式在答案中对应部分引用上下文。如果一句话源自多个上下文，请列出所有相关的引用编号，例如[citation:3][citation:5]，不要将引用集中在最后返回，而是在答案对应部分列出。除非是代码、特定的名称或引用编号，答案的语言应与问题相同。以下是上下文的内容集： ''' + "\n\n" + "```" ref_index = 1 for ref_text in ref_content: prompts = prompts + "\n\n" + " [citation:{}] ".format(str(ref_index)) + ref_text ref_index += 1 if len(prompts) >= limit_len: prompts = prompts[0:limit_len] prompts = prompts + ''' ``` 记住，不要一字不差的重复上下文内容. 回答必须使用简体中文，如果回答很长，请尽量结构化、分段落总结。请按照引用编号[citation:x]的格式在答案中对应部分引用上下文。如果一句话源自多个上下文，请列出所有相关的引用编号，例如[citation:3][citation:5]，不要将引用集中在最后返回，而是在答案对应部分列出。下面是用户问题： ''' + question else: prompts = ''' You are a large language AI assistant develop by nash_su. You are given a user question, and please write clean, concise and accurate answer to the question. You will be given a set of related contexts to the question, each starting with a reference number like [[citation:x]], where x is a number. Please use the context and cite the context at the end of each sentence if applicable. Your answer must be correct, accurate and written by an expert using an unbiased and professional tone. Please limit to 1024 tokens. Do not give any information that is not related to the question, and do not repeat. Say "information is missing on" followed by the related topic, if the given context do not provide sufficient information. Please cite the contexts with the reference numbers, in the format [citation:x]. If a sentence comes from multiple contexts, please list all applicable citations, like [citation:3][citation:5]. Other than code and specific names and citations, your answer must be written in the same language as the question. Here are the set of contexts: ''' + "\n\n" + "```" ref_index = 1 for ref_text in ref_content: prompts = prompts + "\n\n" + " [citation:{}] ".format(str(ref_index)) + ref_text ref_index += 1 if len(prompts) >= limit_len: prompts = prompts[0:limit_len] prompts = prompts + ''' ``` Above is the reference contexts. Remember, don't repeat the context word for word. Answer in ''' + answer_language + '''. If the response is lengthy, structure it in paragraphs and summarize where possible. Cite the context using the format [citation:x] where x is the reference number. If a sentence originates from multiple contexts, list all relevant citation numbers, like [citation:3][citation:5]. Don't cluster the citations at the end but include them in the answer where they correspond. Remember, don't blindly repeat the contexts verbatim. And here is the user question: ''' + question else: prompts = question if debug: print(prompts) print("总长度："+ str(len(prompts))) return prompts def chat(prompt, model:str,llm_auth_token:str,llm_base_url:str,using_custom_llm=False,stream=True, debug=False): openai.base_url = "http://127.0.0.1:3040/v1/" if model == "gpt3.5": openai.base_url = "http://llm-freegpt35:3040/v1/" if model == "kimi": openai.base_url = "http://llm-kimi:8000/v1/" if model == "glm4": openai.base_url = "http://llm-glm4:8000/v1/" if model == "qwen": openai.base_url = "http://llm-qwen:8000/v1/" if llm_auth_token == '': llm_auth_token = "CUSTOM" openai.api_key = llm_auth_token if using_custom_llm: openai.base_url = llm_base_url openai.api_key = llm_auth_token total_content = "" for chunk in openai.chat.completions.create( model=model, messages=[{ "role": "user", "content": prompt }], stream=True, max_tokens=1024,temperature=0.2 ): stream_resp = chunk.dict() token = stream_resp["choices"][0]["delta"].get("content", "") if token: total_content += token yield token if debug: print(total_content) def ask_internet(query:str, debug=False): content_list = search_web_ref(query,debug=debug) if debug: print(content_list) prompt = gen_prompt(query,content_list,context_length_limit=6000,debug=debug) total_token = "" for token in chat(prompt=prompt): # for token in daxianggpt.chat(prompt=prompt): if token: total_token += token yield token yield "\n\n" # 是否返回参考资料 if True: yield "---" yield "\n" yield "参考资料:\n" count = 1 for url_content in content_list: url = url_content.get('url') yield "*[{}. {}]({})*".format(str(count),url,url ) yield "\n" count += 1 ================================================ FILE: requirements.txt ================================================ annotated-types==0.6.0 anyio==4.3.0 certifi==2024.2.2 charset-normalizer==3.3.2 click==8.1.7 courlan==1.0.0 dateparser==1.2.0 distro==1.9.0 exceptiongroup==1.2.0 fastapi==0.110.1 filelock==3.13.3 h11==0.14.0 htmldate==1.8.0 httpcore==1.0.5 httpx==0.27.0 idna==3.6 jusText==3.0.0 langcodes==3.3.0 lxml==5.1.1 openai==1.16.2 pydantic==2.6.4 pydantic_core==2.16.3 python-dateutil==2.9.0.post0 pytz==2024.1 regex==2023.12.25 requests==2.31.0 requests-file==2.0.0 six==1.16.0 sniffio==1.3.1 sse-starlette==2.0.0 starlette==0.37.2 tld==0.13 tldextract==5.1.2 tqdm==4.66.2 trafilatura==1.8.1 typing_extensions==4.10.0 tzlocal==5.2 urllib3==2.2.1 uvicorn==0.29.0 ================================================ FILE: searxng/settings.yml ================================================ general: # Debug mode, only for development. Is overwritten by ${SEARXNG_DEBUG} debug: false # displayed name instance_name: "free_ask_internet" # For example: https://example.com/privacy privacypolicy_url: false # use true to use your own donation page written in searx/info/en/donate.md # use false to disable the donation link donation_url: false # mailto:contact@example.com contact_url: false # record stats enable_metrics: true brand: new_issue_url: https://github.com/searxng/searxng/issues/new docs_url: https://docs.searxng.org/ public_instances: https://searx.space wiki_url: https://github.com/searxng/searxng/wiki issue_url: https://github.com/searxng/searxng/issues # custom: # maintainer: "Jon Doe" # # Custom entries in the footer: [title]: [link] # links: # Uptime: https://uptime.searxng.org/history/darmarit-org # About: "https://searxng.org" search: # Filter results. 0: None, 1: Moderate, 2: Strict safe_search: 0 # Existing autocomplete backends: "dbpedia", "duckduckgo", "google", "yandex", "mwmbl", # "seznam", "startpage", "stract", "swisscows", "qwant", "wikipedia" - leave blank to turn it off # by default. autocomplete: "" # minimun characters to type before autocompleter starts # Default search language - leave blank to detect from browser information or # use codes from 'languages.py' default_lang: "auto" ban_time_on_fail: 1 max_ban_time_on_fail: 120 suspended_times: SearxEngineAccessDenied: 86400 SearxEngineCaptcha: 86400 SearxEngineTooManyRequests: 3600 cf_SearxEngineCaptcha: 1296000 cf_SearxEngineAccessDenied: 86400 recaptcha_SearxEngineCaptcha: 604800 formats: - html - json server: # Is overwritten by ${SEARXNG_PORT} and ${SEARXNG_BIND_ADDRESS} port: 8080 bind_address: "0.0.0.0" # public URL of the instance, to ensure correct inbound links. Is overwritten # by ${SEARXNG_URL}. base_url: http://localhost:/ # "http://example.com/location" limiter: false # rate limit the number of request on the instance, block some bots public_instance: false # enable features designed only for public instances # If your instance owns a /etc/searxng/settings.yml file, then set the following # values there. secret_key: "f884dadd9e6b23ea6c073a031b18daa0288ff1d82a1c5514089f0ecb32ac7a54" # Is overwritten by ${SEARXNG_SECRET} # Proxying image results through searx image_proxy: false # 1.0 and 1.1 are supported http_protocol_version: "1.0" # POST queries are more secure as they don't show up in history but may cause # problems when using Firefox containers method: "POST" default_http_headers: X-Content-Type-Options: nosniff X-Download-Options: noopen X-Robots-Tag: noindex, nofollow Referrer-Policy: no-referrer redis: # URL to connect redis database. Is overwritten by ${SEARXNG_REDIS_URL}. # https://docs.searxng.org/admin/settings/settings_redis.html#settings-redis url: false ui: # Custom static path - leave it blank if you didn't change static_path: "" static_use_hash: false # Custom templates path - leave it blank if you didn't change templates_path: "" # query_in_title: When true, the result page's titles contains the query # it decreases the privacy, since the browser can records the page titles. query_in_title: false # infinite_scroll: When true, automatically loads the next page when scrolling to bottom of the current page. infinite_scroll: false # ui theme default_theme: simple # center the results ? center_alignment: false # URL prefix of the internet archive, don't forget trailing slash (if needed). # cache_url: "https://webcache.googleusercontent.com/search?q=cache:" # Default interface locale - leave blank to detect from browser information or # use codes from the 'locales' config section default_locale: "" # Open result links in a new tab by default # results_on_new_tab: false theme_args: # style of simple theme: auto, light, dark simple_style: auto # Perform search immediately if a category selected. # Disable to select multiple categories at once and start the search manually. search_on_category_select: true # Hotkeys: default or vim hotkeys: default # Lock arbitrary settings on the preferences page. To find the ID of the user # setting you want to lock, check the ID of the form on the page "preferences". # # preferences: # lock: # - language # - autocomplete # - method # - query_in_title # searx supports result proxification using an external service: # https://github.com/asciimoo/morty uncomment below section if you have running # morty proxy the key is base64 encoded (keep the !!binary notation) # Note: since commit af77ec3, morty accepts a base64 encoded key. # # result_proxy: # url: http://127.0.0.1:3000/ # # the key is a base64 encoded string, the YAML !!binary prefix is optional # key: !!binary "your_morty_proxy_key" # # [true|false] enable the "proxy" button next to each result # proxify_results: true # communication with search engines # outgoing: # default timeout in seconds, can be override by engine request_timeout: 3.0 # the maximum timeout in seconds # max_request_timeout: 10.0 # suffix of searx_useragent, could contain information like an email address # to the administrator useragent_suffix: "" # The maximum number of concurrent connections that may be established. pool_connections: 100 # Allow the connection pool to maintain keep-alive connections below this # point. pool_maxsize: 20 # See https://www.python-httpx.org/http2/ enable_http2: true # uncomment below section if you want to use a custom server certificate # see https://www.python-httpx.org/advanced/#changing-the-verification-defaults # and https://www.python-httpx.org/compatibility/#ssl-configuration # verify: ~/.mitmproxy/mitmproxy-ca-cert.cer # # uncomment below section if you want to use a proxyq see: SOCKS proxies # https://2.python-requests.org/en/latest/user/advanced/#proxies # are also supported: see # https://2.python-requests.org/en/latest/user/advanced/#socks # # proxies: # all://: # - http://proxy1:8080 # - http://proxy2:8080 # # using_tor_proxy: true # # Extra seconds to add in order to account for the time taken by the proxy # # extra_proxy_timeout: 10 # # uncomment below section only if you have more than one network interface # which can be the source of outgoing search requests # # source_ips: # - 1.1.1.1 # - 1.1.1.2 # - fe80::/126 # External plugin configuration, for more details see # https://docs.searxng.org/dev/plugins.html # # plugins: # - plugin1 # - plugin2 # - ... # Comment or un-comment plugin to activate / deactivate by default. # # enabled_plugins: # # these plugins are enabled if nothing is configured .. # - 'Hash plugin' # - 'Self Information' # - 'Tracker URL remover' # - 'Ahmia blacklist' # activation depends on outgoing.using_tor_proxy # # these plugins are disabled if nothing is configured .. # - 'Hostname replace' # see hostname_replace configuration below # - 'Open Access DOI rewrite' # - 'Tor check plugin' # # Read the docs before activate: auto-detection of the language could be # # detrimental to users expectations / users can activate the plugin in the # # preferences if they want. # - 'Autodetect search language' # Configuration of the "Hostname replace" plugin: # # hostname_replace: # '(.*\.)?youtube\.com$': 'invidious.example.com' # '(.*\.)?youtu\.be$': 'invidious.example.com' # '(.*\.)?youtube-noocookie\.com$': 'yotter.example.com' # '(.*\.)?reddit\.com$': 'teddit.example.com' # '(.*\.)?redd\.it$': 'teddit.example.com' # '(www\.)?twitter\.com$': 'nitter.example.com' # # to remove matching host names from result list, set value to false # 'spam\.example\.com': false checker: # disable checker when in debug mode off_when_debug: true # use "scheduling: false" to disable scheduling # scheduling: interval or int # to activate the scheduler: # * uncomment "scheduling" section # * add "cache2 = name=searxngcache,items=2000,blocks=2000,blocksize=4096,bitmap=1" # to your uwsgi.ini # scheduling: # start_after: [300, 1800] # delay to start the first run of the checker # every: [86400, 90000] # how often the checker runs # additional tests: only for the YAML anchors (see the engines section) # additional_tests: rosebud: &test_rosebud matrix: query: rosebud lang: en result_container: - not_empty - ['one_title_contains', 'citizen kane'] test: - unique_results android: &test_android matrix: query: ['android'] lang: ['en', 'de', 'fr', 'zh-CN'] result_container: - not_empty - ['one_title_contains', 'google'] test: - unique_results # tests: only for the YAML anchors (see the engines section) tests: infobox: &tests_infobox infobox: matrix: query: ["linux", "new york", "bbc"] result_container: - has_infobox categories_as_tabs: general: images: videos: news: map: music: it: science: files: social media: engines: - name: 9gag engine: 9gag shortcut: 9g disabled: true - name: annas archive engine: annas_archive disabled: true shortcut: aa # - name: annas articles # engine: annas_archive # shortcut: aaa # # https://docs.searxng.org/dev/engines/online/annas_archive.html # aa_content: 'magazine' # book_fiction, book_unknown, book_nonfiction, book_comic # aa_ext: 'pdf' # pdf, epub, .. # aa_sort: oldest' # newest, oldest, largest, smallest - name: apk mirror engine: apkmirror timeout: 4.0 shortcut: apkm disabled: true - name: apple app store engine: apple_app_store shortcut: aps disabled: true # Requires Tor - name: ahmia engine: ahmia categories: onions enable_http: true shortcut: ah - name: anaconda engine: xpath paging: true first_page_num: 0 search_url: https://anaconda.org/search?q={query}&page={pageno} results_xpath: //tbody/tr url_xpath: ./td/h5/a[last()]/@href title_xpath: ./td/h5 content_xpath: ./td[h5]/text() categories: it timeout: 6.0 shortcut: conda disabled: true - name: arch linux wiki engine: archlinux shortcut: al - name: artic engine: artic shortcut: arc timeout: 4.0 - name: arxiv engine: arxiv shortcut: arx timeout: 4.0 - name: ask engine: ask shortcut: ask disabled: true # tmp suspended: dh key too small # - name: base # engine: base # shortcut: bs - name: bandcamp engine: bandcamp shortcut: bc categories: music - name: wikipedia engine: wikipedia shortcut: wp # add "list" to the array to get results in the results list display_type: ["infobox"] base_url: 'https://{language}.wikipedia.org/' categories: [general] - name: bilibili engine: bilibili shortcut: bil disabled: true - name: bing engine: bing shortcut: bi disabled: true - name: bing images engine: bing_images shortcut: bii - name: bing news engine: bing_news shortcut: bin - name: bing videos engine: bing_videos shortcut: biv - name: bitbucket engine: xpath paging: true search_url: https://bitbucket.org/repo/all/{pageno}?name={query} url_xpath: //article[@class="repo-summary"]//a[@class="repo-link"]/@href title_xpath: //article[@class="repo-summary"]//a[@class="repo-link"] content_xpath: //article[@class="repo-summary"]/p categories: [it, repos] timeout: 4.0 disabled: true shortcut: bb about: website: https://bitbucket.org/ wikidata_id: Q2493781 official_api_documentation: https://developer.atlassian.com/bitbucket use_official_api: false require_api_key: false results: HTML - name: bpb engine: bpb shortcut: bpb disabled: true - name: btdigg engine: btdigg shortcut: bt disabled: true - name: ccc-tv engine: xpath paging: false search_url: https://media.ccc.de/search/?q={query} url_xpath: //div[@class="caption"]/h3/a/@href title_xpath: //div[@class="caption"]/h3/a/text() content_xpath: //div[@class="caption"]/h4/@title categories: videos disabled: true shortcut: c3tv about: website: https://media.ccc.de/ wikidata_id: Q80729951 official_api_documentation: https://github.com/voc/voctoweb use_official_api: false require_api_key: false results: HTML # We don't set language: de here because media.ccc.de is not just # for a German audience. It contains many English videos and many # German videos have English subtitles. - name: openverse engine: openverse categories: images shortcut: opv - name: chefkoch engine: chefkoch shortcut: chef # to show premium or plus results too: # skip_premium: false # - name: core.ac.uk # engine: core # categories: science # shortcut: cor # # get your API key from: https://core.ac.uk/api-keys/register/ # api_key: 'unset' - name: cppreference engine: cppreference shortcut: cpp paging: false disabled: true - name: crossref engine: crossref shortcut: cr timeout: 30 disabled: true - name: crowdview engine: json_engine shortcut: cv categories: general paging: false search_url: https://crowdview-next-js.onrender.com/api/search-v3?query={query} results_query: results url_query: link title_query: title content_query: snippet disabled: true about: website: https://crowdview.ai/ - name: yep engine: yep shortcut: yep categories: general search_type: web disabled: true - name: yep images engine: yep shortcut: yepi categories: images search_type: images disabled: true - name: yep news engine: yep shortcut: yepn categories: news search_type: news disabled: true - name: curlie engine: xpath shortcut: cl categories: general disabled: true paging: true lang_all: '' search_url: https://curlie.org/search?q={query}&lang={lang}&start={pageno}&stime=92452189 page_size: 20 results_xpath: //div[@id="site-list-content"]/div[@class="site-item"] url_xpath: ./div[@class="title-and-desc"]/a/@href title_xpath: ./div[@class="title-and-desc"]/a/div content_xpath: ./div[@class="title-and-desc"]/div[@class="site-descr"] about: website: https://curlie.org/ wikidata_id: Q60715723 use_official_api: false require_api_key: false results: HTML - name: currency engine: currency_convert categories: general shortcut: cc - name: bahnhof engine: json_engine search_url: https://www.bahnhof.de/api/stations/search/{query} url_prefix: https://www.bahnhof.de/ url_query: slug title_query: name content_query: state shortcut: bf disabled: true about: website: https://www.bahn.de wikidata_id: Q22811603 use_official_api: false require_api_key: false results: JSON language: de tests: bahnhof: matrix: query: berlin lang: en result_container: - not_empty - ['one_title_contains', 'Berlin Hauptbahnhof'] test: - unique_results - name: deezer engine: deezer shortcut: dz disabled: true - name: destatis engine: destatis shortcut: destat disabled: true - name: deviantart engine: deviantart shortcut: da timeout: 3.0 - name: ddg definitions engine: duckduckgo_definitions shortcut: ddd weight: 2 disabled: true tests: *tests_infobox # cloudflare protected # - name: digbt # engine: digbt # shortcut: dbt # timeout: 6.0 # disabled: true - name: docker hub engine: docker_hub shortcut: dh categories: [it, packages] - name: erowid engine: xpath paging: true first_page_num: 0 page_size: 30 search_url: https://www.erowid.org/search.php?q={query}&s={pageno} url_xpath: //dl[@class="results-list"]/dt[@class="result-title"]/a/@href title_xpath: //dl[@class="results-list"]/dt[@class="result-title"]/a/text() content_xpath: //dl[@class="results-list"]/dd[@class="result-details"] categories: [] shortcut: ew disabled: true about: website: https://www.erowid.org/ wikidata_id: Q1430691 official_api_documentation: use_official_api: false require_api_key: false results: HTML # - name: elasticsearch # shortcut: es # engine: elasticsearch # base_url: http://localhost:9200 # username: elastic # password: changeme # index: my-index # # available options: match, simple_query_string, term, terms, custom # query_type: match # # if query_type is set to custom, provide your query here # #custom_query_json: {"query":{"match_all": {}}} # #show_metadata: false # disabled: true - name: wikidata engine: wikidata shortcut: wd timeout: 3.0 weight: 2 # add "list" to the array to get results in the results list display_type: ["infobox"] tests: *tests_infobox categories: [general] - name: duckduckgo engine: duckduckgo shortcut: ddg - name: duckduckgo images engine: duckduckgo_extra categories: [images, web] ddg_category: images shortcut: ddi disabled: true - name: duckduckgo videos engine: duckduckgo_extra categories: [videos, web] ddg_category: videos shortcut: ddv disabled: true - name: duckduckgo news engine: duckduckgo_extra categories: [news, web] ddg_category: news shortcut: ddn disabled: true - name: duckduckgo weather engine: duckduckgo_weather shortcut: ddw disabled: true - name: apple maps engine: apple_maps shortcut: apm disabled: true timeout: 5.0 - name: emojipedia engine: emojipedia timeout: 4.0 shortcut: em disabled: true - name: tineye engine: tineye shortcut: tin timeout: 9.0 disabled: true - name: etymonline engine: xpath paging: true search_url: https://etymonline.com/search?page={pageno}&q={query} url_xpath: //a[contains(@class, "word__name--")]/@href title_xpath: //a[contains(@class, "word__name--")] content_xpath: //section[contains(@class, "word__defination")] first_page_num: 1 shortcut: et categories: [dictionaries] about: website: https://www.etymonline.com/ wikidata_id: Q1188617 official_api_documentation: use_official_api: false require_api_key: false results: HTML # - name: ebay # engine: ebay # shortcut: eb # base_url: 'https://www.ebay.com' # disabled: true # timeout: 5 - name: 1x engine: www1x shortcut: 1x timeout: 3.0 disabled: true - name: fdroid engine: fdroid shortcut: fd disabled: true - name: flickr categories: images shortcut: fl # You can use the engine using the official stable API, but you need an API # key, see: https://www.flickr.com/services/apps/create/ # engine: flickr # api_key: 'apikey' # required! # Or you can use the html non-stable engine, activated by default engine: flickr_noapi - name: free software directory engine: mediawiki shortcut: fsd categories: [it, software wikis] base_url: https://directory.fsf.org/ search_type: title timeout: 5.0 disabled: true about: website: https://directory.fsf.org/ wikidata_id: Q2470288 # - name: freesound # engine: freesound # shortcut: fnd # disabled: true # timeout: 15.0 # API key required, see: https://freesound.org/docs/api/overview.html # api_key: MyAPIkey - name: frinkiac engine: frinkiac shortcut: frk disabled: true - name: fyyd engine: fyyd shortcut: fy timeout: 8.0 disabled: true - name: genius engine: genius shortcut: gen - name: gentoo engine: gentoo shortcut: ge timeout: 10.0 - name: gitlab engine: json_engine paging: true search_url: https://gitlab.com/api/v4/projects?search={query}&page={pageno} url_query: web_url title_query: name_with_namespace content_query: description page_size: 20 categories: [it, repos] shortcut: gl timeout: 10.0 disabled: true about: website: https://about.gitlab.com/ wikidata_id: Q16639197 official_api_documentation: https://docs.gitlab.com/ee/api/ use_official_api: false require_api_key: false results: JSON - name: github engine: github shortcut: gh # This a Gitea service. If you would like to use a different instance, # change codeberg.org to URL of the desired Gitea host. Or you can create a # new engine by copying this and changing the name, shortcut and search_url. - name: codeberg engine: json_engine search_url: https://codeberg.org/api/v1/repos/search?q={query}&limit=10 url_query: html_url title_query: name content_query: description categories: [it, repos] shortcut: cb disabled: true about: website: https://codeberg.org/ wikidata_id: official_api_documentation: https://try.gitea.io/api/swagger use_official_api: false require_api_key: false results: JSON - name: goodreads engine: goodreads shortcut: good timeout: 4.0 disabled: true - name: google engine: google shortcut: go # additional_tests: # android: *test_android - name: google images engine: google_images shortcut: goi # additional_tests: # android: *test_android # dali: # matrix: # query: ['Dali Christ'] # lang: ['en', 'de', 'fr', 'zh-CN'] # result_container: # - ['one_title_contains', 'Salvador'] - name: google news engine: google_news shortcut: gon # additional_tests: # android: *test_android - name: google videos engine: google_videos shortcut: gov # additional_tests: # android: *test_android - name: google scholar engine: google_scholar shortcut: gos - name: google play apps engine: google_play categories: [files, apps] shortcut: gpa play_categ: apps disabled: true - name: google play movies engine: google_play categories: videos shortcut: gpm play_categ: movies disabled: true - name: material icons engine: material_icons categories: images shortcut: mi disabled: true - name: gpodder engine: json_engine shortcut: gpod timeout: 4.0 paging: false search_url: https://gpodder.net/search.json?q={query} url_query: url title_query: title content_query: description page_size: 19 categories: music disabled: true about: website: https://gpodder.net wikidata_id: Q3093354 official_api_documentation: https://gpoddernet.readthedocs.io/en/latest/api/ use_official_api: false requires_api_key: false results: JSON - name: habrahabr engine: xpath paging: true search_url: https://habr.com/en/search/page{pageno}/?q={query} results_xpath: //article[contains(@class, "tm-articles-list__item")] url_xpath: .//a[@class="tm-title__link"]/@href title_xpath: .//a[@class="tm-title__link"] content_xpath: .//div[contains(@class, "article-formatted-body")] categories: it timeout: 4.0 disabled: true shortcut: habr about: website: https://habr.com/ wikidata_id: Q4494434 official_api_documentation: https://habr.com/en/docs/help/api/ use_official_api: false require_api_key: false results: HTML - name: hackernews engine: hackernews shortcut: hn disabled: true - name: hoogle engine: xpath search_url: https://hoogle.haskell.org/?hoogle={query} results_xpath: '//div[@class="result"]' title_xpath: './/div[@class="ans"]//a' url_xpath: './/div[@class="ans"]//a/@href' content_xpath: './/div[@class="from"]' page_size: 20 categories: [it, packages] shortcut: ho about: website: https://hoogle.haskell.org/ wikidata_id: Q34010 official_api_documentation: https://hackage.haskell.org/api use_official_api: false require_api_key: false results: JSON - name: imdb engine: imdb shortcut: imdb timeout: 6.0 disabled: true - name: imgur engine: imgur shortcut: img disabled: true - name: ina engine: ina shortcut: in timeout: 6.0 disabled: true - name: invidious engine: invidious # Instanes will be selected randomly, see https://api.invidious.io/ for # instances that are stable (good uptime) and close to you. base_url: - https://invidious.io.lol - https://invidious.fdn.fr - https://yt.artemislena.eu - https://invidious.tiekoetter.com - https://invidious.flokinet.to - https://vid.puffyan.us - https://invidious.privacydev.net - https://inv.tux.pizza shortcut: iv timeout: 3.0 disabled: true - name: jisho engine: jisho shortcut: js timeout: 3.0 disabled: true - name: kickass engine: kickass base_url: - https://kickasstorrents.to - https://kickasstorrents.cr - https://kickasstorrent.cr - https://kickass.sx - https://kat.am shortcut: kc timeout: 4.0 - name: lemmy communities engine: lemmy lemmy_type: Communities shortcut: leco - name: lemmy users engine: lemmy network: lemmy communities lemmy_type: Users shortcut: leus - name: lemmy posts engine: lemmy network: lemmy communities lemmy_type: Posts shortcut: lepo - name: lemmy comments engine: lemmy network: lemmy communities lemmy_type: Comments shortcut: lecom - name: library genesis engine: xpath # search_url: https://libgen.is/search.php?req={query} search_url: https://libgen.rs/search.php?req={query} url_xpath: //a[contains(@href,"book/index.php?md5")]/@href title_xpath: //a[contains(@href,"book/")]/text()[1] content_xpath: //td/a[1][contains(@href,"=author")]/text() categories: files timeout: 7.0 disabled: true shortcut: lg about: website: https://libgen.fun/ wikidata_id: Q22017206 official_api_documentation: use_official_api: false require_api_key: false results: HTML - name: z-library engine: zlibrary shortcut: zlib categories: files timeout: 7.0 - name: library of congress engine: loc shortcut: loc categories: images - name: lingva engine: lingva shortcut: lv # set lingva instance in url, by default it will use the official instance # url: https://lingva.thedaviddelta.com - name: lobste.rs engine: xpath search_url: https://lobste.rs/search?q={query}&what=stories&order=relevance results_xpath: //li[contains(@class, "story")] url_xpath: .//a[@class="u-url"]/@href title_xpath: .//a[@class="u-url"] content_xpath: .//a[@class="domain"] categories: it shortcut: lo timeout: 5.0 disabled: true about: website: https://lobste.rs/ wikidata_id: Q60762874 official_api_documentation: use_official_api: false require_api_key: false results: HTML - name: azlyrics shortcut: lyrics engine: xpath timeout: 4.0 disabled: true categories: [music, lyrics] paging: true search_url: https://search.azlyrics.com/search.php?q={query}&w=lyrics&p={pageno} url_xpath: //td[@class="text-left visitedlyr"]/a/@href title_xpath: //span/b/text() content_xpath: //td[@class="text-left visitedlyr"]/a/small about: website: https://azlyrics.com wikidata_id: Q66372542 official_api_documentation: use_official_api: false require_api_key: false results: HTML - name: mastodon users engine: mastodon mastodon_type: accounts base_url: https://mastodon.social shortcut: mau - name: mastodon hashtags engine: mastodon mastodon_type: hashtags base_url: https://mastodon.social shortcut: mah # - name: matrixrooms # engine: mrs # # https://docs.searxng.org/dev/engines/online/mrs.html # # base_url: https://mrs-api-host # shortcut: mtrx # disabled: true - name: mdn shortcut: mdn engine: json_engine categories: [it] paging: true search_url: https://developer.mozilla.org/api/v1/search?q={query}&page={pageno} results_query: documents url_query: mdn_url url_prefix: https://developer.mozilla.org title_query: title content_query: summary about: website: https://developer.mozilla.org wikidata_id: Q3273508 official_api_documentation: null use_official_api: false require_api_key: false results: JSON - name: metacpan engine: metacpan shortcut: cpan disabled: true number_of_results: 20 # - name: meilisearch # engine: meilisearch # shortcut: mes # enable_http: true # base_url: http://localhost:7700 # index: my-index - name: mixcloud engine: mixcloud shortcut: mc # MongoDB engine # Required dependency: pymongo # - name: mymongo # engine: mongodb # shortcut: md # exact_match_only: false # host: '127.0.0.1' # port: 27017 # enable_http: true # results_per_page: 20 # database: 'business' # collection: 'reviews' # name of the db collection # key: 'name' # key in the collection to search for - name: mozhi engine: mozhi base_url: - https://mozhi.aryak.me - https://translate.bus-hit.me - https://nyc1.mz.ggtyler.dev # mozhi_engine: google - see https://mozhi.aryak.me for supported engines timeout: 4.0 shortcut: mz disabled: true - name: mwmbl engine: mwmbl # api_url: https://api.mwmbl.org shortcut: mwm disabled: true - name: npm engine: npm shortcut: npm timeout: 5.0 disabled: true - name: nyaa engine: nyaa shortcut: nt disabled: true - name: mankier engine: json_engine search_url: https://www.mankier.com/api/v2/mans/?q={query} results_query: results url_query: url title_query: name content_query: description categories: it shortcut: man about: website: https://www.mankier.com/ official_api_documentation: https://www.mankier.com/api use_official_api: true require_api_key: false results: JSON # read https://docs.searxng.org/dev/engines/online/mullvad_leta.html # - name: mullvadleta # engine: mullvad_leta # use_cache: true # Only 100 non-cache searches per day, suggested only for private instances # search_url: https://leta.mullvad.net # categories: [general, web] # shortcut: ml - name: odysee engine: odysee shortcut: od disabled: true - name: openairedatasets engine: json_engine paging: true search_url: https://api.openaire.eu/search/datasets?format=json&page={pageno}&size=10&title={query} results_query: response/results/result url_query: metadata/oaf:entity/oaf:result/children/instance/webresource/url/$ title_query: metadata/oaf:entity/oaf:result/title/$ content_query: metadata/oaf:entity/oaf:result/description/$ content_html_to_text: true categories: "science" shortcut: oad timeout: 5.0 about: website: https://www.openaire.eu/ wikidata_id: Q25106053 official_api_documentation: https://api.openaire.eu/ use_official_api: false require_api_key: false results: JSON - name: openairepublications engine: json_engine paging: true search_url: https://api.openaire.eu/search/publications?format=json&page={pageno}&size=10&title={query} results_query: response/results/result url_query: metadata/oaf:entity/oaf:result/children/instance/webresource/url/$ title_query: metadata/oaf:entity/oaf:result/title/$ content_query: metadata/oaf:entity/oaf:result/description/$ content_html_to_text: true categories: science shortcut: oap timeout: 5.0 about: website: https://www.openaire.eu/ wikidata_id: Q25106053 official_api_documentation: https://api.openaire.eu/ use_official_api: false require_api_key: false results: JSON # - name: opensemanticsearch # engine: opensemantic # shortcut: oss # base_url: 'http://localhost:8983/solr/opensemanticsearch/' - name: openstreetmap engine: openstreetmap shortcut: osm - name: openrepos engine: xpath paging: true search_url: https://openrepos.net/search/node/{query}?page={pageno} url_xpath: //li[@class="search-result"]//h3[@class="title"]/a/@href title_xpath: //li[@class="search-result"]//h3[@class="title"]/a content_xpath: //li[@class="search-result"]//div[@class="search-snippet-info"]//p[@class="search-snippet"] categories: files timeout: 4.0 disabled: true shortcut: or about: website: https://openrepos.net/ wikidata_id: official_api_documentation: use_official_api: false require_api_key: false results: HTML - name: packagist engine: json_engine paging: true search_url: https://packagist.org/search.json?q={query}&page={pageno} results_query: results url_query: url title_query: name content_query: description categories: [it, packages] disabled: true timeout: 5.0 shortcut: pack about: website: https://packagist.org wikidata_id: Q108311377 official_api_documentation: https://packagist.org/apidoc use_official_api: true require_api_key: false results: JSON - name: pdbe engine: pdbe shortcut: pdb # Hide obsolete PDB entries. Default is not to hide obsolete structures # hide_obsolete: false - name: photon engine: photon shortcut: ph - name: pinterest engine: pinterest shortcut: pin - name: piped engine: piped shortcut: ppd categories: videos piped_filter: videos timeout: 3.0 # URL to use as link and for embeds frontend_url: https://srv.piped.video # Instance will be selected randomly, for more see https://piped-instances.kavin.rocks/ backend_url: - https://pipedapi.kavin.rocks - https://pipedapi-libre.kavin.rocks - https://pipedapi.adminforge.de - name: piped.music engine: piped network: piped shortcut: ppdm categories: music piped_filter: music_songs timeout: 3.0 - name: piratebay engine: piratebay shortcut: tpb # You may need to change this URL to a proxy if piratebay is blocked in your # country url: https://thepiratebay.org/ timeout: 3.0 - name: pixiv shortcut: pv engine: pixiv disabled: true inactive: true pixiv_image_proxies: - pximg.example.org # A proxy is required to load the images. Hosting an image proxy server # for Pixiv: # --> https://codeberg.org/VnPower/PixivFE/wiki/Hosting-an-image-proxy-server-for-Pixiv # Proxies from public instances. Ask the public instances owners if they # agree to receive traffic from SearXNG! # --> https://codeberg.org/VnPower/PixivFE#instances # --> https://github.com/searxng/searxng/pull/3192#issuecomment-1941095047 # image proxy of https://pixiv.cat # - https://i.pixiv.cat # image proxy of https://www.pixiv.pics # - https://pximg.cocomi.eu.org # image proxy of https://pixivfe.exozy.me # - https://pximg.exozy.me # image proxy of https://pixivfe.ducks.party # - https://pixiv.ducks.party # image proxy of https://pixiv.perennialte.ch # - https://pximg.perennialte.ch - name: podcastindex engine: podcastindex shortcut: podcast # Required dependency: psychopg2 # - name: postgresql # engine: postgresql # database: postgres # username: postgres # password: postgres # limit: 10 # query_str: 'SELECT * from my_table WHERE my_column = %(query)s' # shortcut : psql - name: presearch engine: presearch search_type: search categories: [general, web] shortcut: ps timeout: 4.0 disabled: true - name: presearch images engine: presearch network: presearch search_type: images categories: [images, web] timeout: 4.0 shortcut: psimg disabled: true - name: presearch videos engine: presearch network: presearch search_type: videos categories: [general, web] timeout: 4.0 shortcut: psvid disabled: true - name: presearch news engine: presearch network: presearch search_type: news categories: [news, web] timeout: 4.0 shortcut: psnews disabled: true - name: pub.dev engine: xpath shortcut: pd search_url: https://pub.dev/packages?q={query}&page={pageno} paging: true results_xpath: //div[contains(@class,"packages-item")] url_xpath: ./div/h3/a/@href title_xpath: ./div/h3/a content_xpath: ./div/div/div[contains(@class,"packages-description")]/span categories: [packages, it] timeout: 3.0 disabled: true first_page_num: 1 about: website: https://pub.dev/ official_api_documentation: https://pub.dev/help/api use_official_api: false require_api_key: false results: HTML - name: pubmed engine: pubmed shortcut: pub timeout: 3.0 - name: pypi shortcut: pypi engine: pypi - name: qwant qwant_categ: web engine: qwant shortcut: qw categories: [general, web] additional_tests: rosebud: *test_rosebud - name: qwant news qwant_categ: news engine: qwant shortcut: qwn categories: news network: qwant - name: qwant images qwant_categ: images engine: qwant shortcut: qwi categories: [images, web] network: qwant - name: qwant videos qwant_categ: videos engine: qwant shortcut: qwv categories: [videos, web] network: qwant # - name: library # engine: recoll # shortcut: lib # base_url: 'https://recoll.example.org/' # search_dir: '' # mount_prefix: /export # dl_prefix: 'https://download.example.org' # timeout: 30.0 # categories: files # disabled: true # - name: recoll library reference # engine: recoll # base_url: 'https://recoll.example.org/' # search_dir: reference # mount_prefix: /export # dl_prefix: 'https://download.example.org' # shortcut: libr # timeout: 30.0 # categories: files # disabled: true - name: radio browser engine: radio_browser shortcut: rb - name: reddit engine: reddit shortcut: re page_size: 25 - name: rottentomatoes engine: rottentomatoes shortcut: rt disabled: true # Required dependency: redis # - name: myredis # shortcut : rds # engine: redis_server # exact_match_only: false # host: '127.0.0.1' # port: 6379 # enable_http: true # password: '' # db: 0 # tmp suspended: bad certificate # - name: scanr structures # shortcut: scs # engine: scanr_structures # disabled: true - name: sepiasearch engine: sepiasearch shortcut: sep - name: soundcloud engine: soundcloud shortcut: sc - name: stackoverflow engine: stackexchange shortcut: st api_site: 'stackoverflow' categories: [it, q&a] - name: askubuntu engine: stackexchange shortcut: ubuntu api_site: 'askubuntu' categories: [it, q&a] - name: internetarchivescholar engine: internet_archive_scholar shortcut: ias timeout: 15.0 - name: superuser engine: stackexchange shortcut: su api_site: 'superuser' categories: [it, q&a] - name: searchcode code engine: searchcode_code shortcut: scc disabled: true # - name: searx # engine: searx_engine # shortcut: se # instance_urls : # - http://127.0.0.1:8888/ # - ... # disabled: true - name: semantic scholar engine: semantic_scholar disabled: true shortcut: se # Spotify needs API credentials # - name: spotify # engine: spotify # shortcut: stf # api_client_id: ******* # api_client_secret: ******* # - name: solr # engine: solr # shortcut: slr # base_url: http://localhost:8983 # collection: collection_name # sort: '' # sorting: asc or desc # field_list: '' # comma separated list of field names to display on the UI # default_fields: '' # default field to query # query_fields: '' # query fields # enable_http: true # - name: springer nature # engine: springer # # get your API key from: https://dev.springernature.com/signup # # working API key, for test & debug: "a69685087d07eca9f13db62f65b8f601" # api_key: 'unset' # shortcut: springer # timeout: 15.0 - name: startpage engine: startpage shortcut: sp timeout: 6.0 disabled: true additional_tests: rosebud: *test_rosebud - name: tokyotoshokan engine: tokyotoshokan shortcut: tt timeout: 6.0 disabled: true - name: solidtorrents engine: solidtorrents shortcut: solid timeout: 4.0 base_url: - https://solidtorrents.to - https://bitsearch.to # For this demo of the sqlite engine download: # https://liste.mediathekview.de/filmliste-v2.db.bz2 # and unpack into searx/data/filmliste-v2.db # Query to test: "!demo concert" # # - name: demo # engine: sqlite # shortcut: demo # categories: general # result_template: default.html # database: searx/data/filmliste-v2.db # query_str: >- # SELECT title || ' (' || time(duration, 'unixepoch') || ')' AS title, # COALESCE( NULLIF(url_video_hd,''), NULLIF(url_video_sd,''), url_video) AS url, # description AS content # FROM film # WHERE title LIKE :wildcard OR description LIKE :wildcard # ORDER BY duration DESC - name: tagesschau engine: tagesschau # when set to false, display URLs from Tagesschau, and not the actual source # (e.g. NDR, WDR, SWR, HR, ...) use_source_url: true shortcut: ts disabled: true - name: tmdb engine: xpath paging: true categories: movies search_url: https://www.themoviedb.org/search?page={pageno}&query={query} results_xpath: //div[contains(@class,"movie") or contains(@class,"tv")]//div[contains(@class,"card")] url_xpath: .//div[contains(@class,"poster")]/a/@href thumbnail_xpath: .//img/@src title_xpath: .//div[contains(@class,"title")]//h2 content_xpath: .//div[contains(@class,"overview")] shortcut: tm disabled: true # Requires Tor - name: torch engine: xpath paging: true search_url: http://xmh57jrknzkhv6y3ls3ubitzfqnkrwxhopf5aygthi7d6rplyvk3noyd.onion/cgi-bin/omega/omega?P={query}&DEFAULTOP=and results_xpath: //table//tr url_xpath: ./td[2]/a title_xpath: ./td[2]/b content_xpath: ./td[2]/small categories: onions enable_http: true shortcut: tch # torznab engine lets you query any torznab compatible indexer. Using this # engine in combination with Jackett opens the possibility to query a lot of # public and private indexers directly from SearXNG. More details at: # https://docs.searxng.org/dev/engines/online/torznab.html # # - name: Torznab EZTV # engine: torznab # shortcut: eztv # base_url: http://localhost:9117/api/v2.0/indexers/eztv/results/torznab # enable_http: true # if using localhost # api_key: xxxxxxxxxxxxxxx # show_magnet_links: true # show_torrent_files: false # # https://github.com/Jackett/Jackett/wiki/Jackett-Categories # torznab_categories: # optional # - 2000 # - 5000 # tmp suspended - too slow, too many errors # - name: urbandictionary # engine : xpath # search_url : https://www.urbandictionary.com/define.php?term={query} # url_xpath : //*[@class="word"]/@href # title_xpath : //*[@class="def-header"] # content_xpath: //*[@class="meaning"] # shortcut: ud - name: unsplash engine: unsplash shortcut: us - name: yandex music engine: yandex_music shortcut: ydm disabled: true # https://yandex.com/support/music/access.html inactive: true - name: yahoo engine: yahoo shortcut: yh disabled: true - name: yahoo news engine: yahoo_news shortcut: yhn - name: youtube shortcut: yt # You can use the engine using the official stable API, but you need an API # key See: https://console.developers.google.com/project # # engine: youtube_api # api_key: 'apikey' # required! # # Or you can use the html non-stable engine, activated by default engine: youtube_noapi - name: dailymotion engine: dailymotion shortcut: dm - name: vimeo engine: vimeo shortcut: vm - name: wiby engine: json_engine paging: true search_url: https://wiby.me/json/?q={query}&p={pageno} url_query: URL title_query: Title content_query: Snippet categories: [general, web] shortcut: wib disabled: true about: website: https://wiby.me/ - name: alexandria engine: json_engine shortcut: alx categories: general paging: true search_url: https://api.alexandria.org/?a=1&q={query}&p={pageno} results_query: results title_query: title url_query: url content_query: snippet timeout: 1.5 disabled: true about: website: https://alexandria.org/ official_api_documentation: https://github.com/alexandria-org/alexandria-api/raw/master/README.md use_official_api: true require_api_key: false results: JSON - name: wikibooks engine: mediawiki weight: 0.5 shortcut: wb categories: [general, wikimedia] base_url: "https://{language}.wikibooks.org/" search_type: text disabled: true about: website: https://www.wikibooks.org/ wikidata_id: Q367 - name: wikinews engine: mediawiki shortcut: wn categories: [news, wikimedia] base_url: "https://{language}.wikinews.org/" search_type: text srsort: create_timestamp_desc about: website: https://www.wikinews.org/ wikidata_id: Q964 - name: wikiquote engine: mediawiki weight: 0.5 shortcut: wq categories: [general, wikimedia] base_url: "https://{language}.wikiquote.org/" search_type: text disabled: true additional_tests: rosebud: *test_rosebud about: website: https://www.wikiquote.org/ wikidata_id: Q369 - name: wikisource engine: mediawiki weight: 0.5 shortcut: ws categories: [general, wikimedia] base_url: "https://{language}.wikisource.org/" search_type: text disabled: true about: website: https://www.wikisource.org/ wikidata_id: Q263 - name: wikispecies engine: mediawiki shortcut: wsp categories: [general, science, wikimedia] base_url: "https://species.wikimedia.org/" search_type: text disabled: true about: website: https://species.wikimedia.org/ wikidata_id: Q13679 tests: wikispecies: matrix: query: "Campbell, L.I. et al. 2011: MicroRNAs" lang: en result_container: - not_empty - ['one_title_contains', 'Tardigrada'] test: - unique_results - name: wiktionary engine: mediawiki shortcut: wt categories: [dictionaries, wikimedia] base_url: "https://{language}.wiktionary.org/" search_type: text about: website: https://www.wiktionary.org/ wikidata_id: Q151 - name: wikiversity engine: mediawiki weight: 0.5 shortcut: wv categories: [general, wikimedia] base_url: "https://{language}.wikiversity.org/" search_type: text disabled: true about: website: https://www.wikiversity.org/ wikidata_id: Q370 - name: wikivoyage engine: mediawiki weight: 0.5 shortcut: wy categories: [general, wikimedia] base_url: "https://{language}.wikivoyage.org/" search_type: text disabled: true about: website: https://www.wikivoyage.org/ wikidata_id: Q373 - name: wikicommons.images engine: wikicommons shortcut: wc categories: images number_of_results: 10 - name: wolframalpha shortcut: wa # You can use the engine using the official stable API, but you need an API # key. See: https://products.wolframalpha.com/api/ # # engine: wolframalpha_api # api_key: '' # # Or you can use the html non-stable engine, activated by default engine: wolframalpha_noapi timeout: 6.0 categories: general disabled: true - name: dictzone engine: dictzone shortcut: dc - name: mymemory translated engine: translated shortcut: tl timeout: 5.0 # You can use without an API key, but you are limited to 1000 words/day # See: https://mymemory.translated.net/doc/usagelimits.php # api_key: '' # Required dependency: mysql-connector-python # - name: mysql # engine: mysql_server # database: mydatabase # username: user # password: pass # limit: 10 # query_str: 'SELECT * from mytable WHERE fieldname=%(query)s' # shortcut: mysql - name: 1337x engine: 1337x shortcut: 1337x disabled: true - name: duden engine: duden shortcut: du disabled: true - name: seznam shortcut: szn engine: seznam disabled: true # - name: deepl # engine: deepl # shortcut: dpl # # You can use the engine using the official stable API, but you need an API key # # See: https://www.deepl.com/pro-api?cta=header-pro-api # api_key: '' # required! # timeout: 5.0 # disabled: true - name: mojeek shortcut: mjk engine: xpath paging: true categories: [general, web] search_url: https://www.mojeek.com/search?q={query}&s={pageno}&lang={lang}&lb={lang} results_xpath: //ul[@class="results-standard"]/li/a[@class="ob"] url_xpath: ./@href title_xpath: ../h2/a content_xpath: ..//p[@class="s"] suggestion_xpath: //div[@class="top-info"]/p[@class="top-info spell"]/em/a first_page_num: 0 page_size: 10 max_page: 100 disabled: true about: website: https://www.mojeek.com/ wikidata_id: Q60747299 official_api_documentation: https://www.mojeek.com/services/api.html/ use_official_api: false require_api_key: false results: HTML - name: moviepilot engine: moviepilot shortcut: mp disabled: true - name: naver shortcut: nvr categories: [general, web] engine: xpath paging: true search_url: https://search.naver.com/search.naver?where=webkr&sm=osp_hty&ie=UTF-8&query={query}&start={pageno} url_xpath: //a[@class="link_tit"]/@href title_xpath: //a[@class="link_tit"] content_xpath: //div[@class="total_dsc_wrap"]/a first_page_num: 1 page_size: 10 disabled: true about: website: https://www.naver.com/ wikidata_id: Q485639 official_api_documentation: https://developers.naver.com/docs/nmt/examples/ use_official_api: false require_api_key: false results: HTML language: ko - name: rubygems shortcut: rbg engine: xpath paging: true search_url: https://rubygems.org/search?page={pageno}&query={query} results_xpath: /html/body/main/div/a[@class="gems__gem"] url_xpath: ./@href title_xpath: ./span/h2 content_xpath: ./span/p suggestion_xpath: /html/body/main/div/div[@class="search__suggestions"]/p/a first_page_num: 1 categories: [it, packages] disabled: true about: website: https://rubygems.org/ wikidata_id: Q1853420 official_api_documentation: https://guides.rubygems.org/rubygems-org-api/ use_official_api: false require_api_key: false results: HTML - name: peertube engine: peertube shortcut: ptb paging: true # alternatives see: https://instances.joinpeertube.org/instances # base_url: https://tube.4aem.com categories: videos disabled: true timeout: 6.0 - name: mediathekviewweb engine: mediathekviewweb shortcut: mvw disabled: true - name: yacy engine: yacy categories: general search_type: text base_url: https://yacy.searchlab.eu shortcut: ya disabled: true # required if you aren't using HTTPS for your local yacy instance # https://docs.searxng.org/dev/engines/online/yacy.html # enable_http: true # timeout: 3.0 # search_mode: 'global' - name: yacy images engine: yacy categories: images search_type: image base_url: https://yacy.searchlab.eu shortcut: yai disabled: true - name: rumble engine: rumble shortcut: ru base_url: https://rumble.com/ paging: true categories: videos disabled: true - name: livespace engine: livespace shortcut: ls categories: videos disabled: true timeout: 5.0 - name: wordnik engine: wordnik shortcut: def base_url: https://www.wordnik.com/ categories: [dictionaries] timeout: 5.0 - name: woxikon.de synonyme engine: xpath shortcut: woxi categories: [dictionaries] timeout: 5.0 disabled: true search_url: https://synonyme.woxikon.de/synonyme/{query}.php url_xpath: //div[@class="upper-synonyms"]/a/@href content_xpath: //div[@class="synonyms-list-group"] title_xpath: //div[@class="upper-synonyms"]/a no_result_for_http_status: [404] about: website: https://www.woxikon.de/ wikidata_id: # No Wikidata ID use_official_api: false require_api_key: false results: HTML language: de - name: seekr news engine: seekr shortcut: senews categories: news seekr_category: news disabled: true - name: seekr images engine: seekr network: seekr news shortcut: seimg categories: images seekr_category: images disabled: true - name: seekr videos engine: seekr network: seekr news shortcut: sevid categories: videos seekr_category: videos disabled: true - name: sjp.pwn engine: sjp shortcut: sjp base_url: https://sjp.pwn.pl/ timeout: 5.0 disabled: true - name: stract engine: stract shortcut: str disabled: true - name: svgrepo engine: svgrepo shortcut: svg timeout: 10.0 disabled: true - name: tootfinder engine: tootfinder shortcut: toot - name: voidlinux engine: voidlinux shortcut: void disabled: true - name: wallhaven engine: wallhaven # api_key: abcdefghijklmnopqrstuvwxyz shortcut: wh # wikimini: online encyclopedia for children # The fulltext and title parameter is necessary for Wikimini because # sometimes it will not show the results and redirect instead - name: wikimini engine: xpath shortcut: wkmn search_url: https://fr.wikimini.org/w/index.php?search={query}&title=Sp%C3%A9cial%3ASearch&fulltext=Search url_xpath: //li/div[@class="mw-search-result-heading"]/a/@href title_xpath: //li//div[@class="mw-search-result-heading"]/a content_xpath: //li/div[@class="searchresult"] categories: general disabled: true about: website: https://wikimini.org/ wikidata_id: Q3568032 use_official_api: false require_api_key: false results: HTML language: fr - name: wttr.in engine: wttr shortcut: wttr timeout: 9.0 - name: yummly engine: yummly shortcut: yum disabled: true - name: brave engine: brave shortcut: br time_range_support: true paging: true categories: [general, web] brave_category: search # brave_spellcheck: true - name: brave.images engine: brave network: brave shortcut: brimg categories: [images, web] brave_category: images - name: brave.videos engine: brave network: brave shortcut: brvid categories: [videos, web] brave_category: videos - name: brave.news engine: brave network: brave shortcut: brnews categories: news brave_category: news # - name: brave.goggles # engine: brave # network: brave # shortcut: brgog # time_range_support: true # paging: true # categories: [general, web] # brave_category: goggles # Goggles: # required! This should be a URL ending in .goggle - name: lib.rs shortcut: lrs engine: lib_rs disabled: true - name: sourcehut shortcut: srht engine: xpath paging: true search_url: https://sr.ht/projects?page={pageno}&search={query} results_xpath: (//div[@class="event-list"])[1]/div[@class="event"] url_xpath: ./h4/a[2]/@href title_xpath: ./h4/a[2] content_xpath: ./p first_page_num: 1 categories: [it, repos] disabled: true about: website: https://sr.ht wikidata_id: Q78514485 official_api_documentation: https://man.sr.ht/ use_official_api: false require_api_key: false results: HTML - name: goo shortcut: goo engine: xpath paging: true search_url: https://search.goo.ne.jp/web.jsp?MT={query}&FR={pageno}0 url_xpath: //div[@class="result"]/p[@class='title fsL1']/a/@href title_xpath: //div[@class="result"]/p[@class='title fsL1']/a content_xpath: //p[contains(@class,'url fsM')]/following-sibling::p first_page_num: 0 categories: [general, web] disabled: true timeout: 4.0 about: website: https://search.goo.ne.jp wikidata_id: Q249044 use_official_api: false require_api_key: false results: HTML language: ja - name: bt4g engine: bt4g shortcut: bt4g - name: pkg.go.dev engine: pkg_go_dev shortcut: pgo disabled: true # Doku engine lets you access to any Doku wiki instance: # A public one or a privete/corporate one. # - name: ubuntuwiki # engine: doku # shortcut: uw # base_url: 'https://doc.ubuntu-fr.org' # Be careful when enabling this engine if you are # running a public instance. Do not expose any sensitive # information. You can restrict access by configuring a list # of access tokens under tokens. # - name: git grep # engine: command # command: ['git', 'grep', '{{QUERY}}'] # shortcut: gg # tokens: [] # disabled: true # delimiter: # chars: ':' # keys: ['filepath', 'code'] # Be careful when enabling this engine if you are # running a public instance. Do not expose any sensitive # information. You can restrict access by configuring a list # of access tokens under tokens. # - name: locate # engine: command # command: ['locate', '{{QUERY}}'] # shortcut: loc # tokens: [] # disabled: true # delimiter: # chars: ' ' # keys: ['line'] # Be careful when enabling this engine if you are # running a public instance. Do not expose any sensitive # information. You can restrict access by configuring a list # of access tokens under tokens. # - name: find # engine: command # command: ['find', '.', '-name', '{{QUERY}}'] # query_type: path # shortcut: fnd # tokens: [] # disabled: true # delimiter: # chars: ' ' # keys: ['line'] # Be careful when enabling this engine if you are # running a public instance. Do not expose any sensitive # information. You can restrict access by configuring a list # of access tokens under tokens. # - name: pattern search in files # engine: command # command: ['fgrep', '{{QUERY}}'] # shortcut: fgr # tokens: [] # disabled: true # delimiter: # chars: ' ' # keys: ['line'] # Be careful when enabling this engine if you are # running a public instance. Do not expose any sensitive # information. You can restrict access by configuring a list # of access tokens under tokens. # - name: regex search in files # engine: command # command: ['grep', '{{QUERY}}'] # shortcut: gr # tokens: [] # disabled: true # delimiter: # chars: ' ' # keys: ['line'] doi_resolvers: oadoi.org: 'https://oadoi.org/' doi.org: 'https://doi.org/' doai.io: 'https://dissem.in/' sci-hub.se: 'https://sci-hub.se/' sci-hub.st: 'https://sci-hub.st/' sci-hub.ru: 'https://sci-hub.ru/' default_doi_resolver: 'oadoi.org' ================================================ FILE: searxng/uwsgi.ini ================================================ [uwsgi] # Who will run the code uid = searxng gid = searxng # Number of workers (usually CPU count) # default value: %k (= number of CPU core, see Dockerfile) workers = %k # Number of threads per worker # default value: 4 (see Dockerfile) threads = 4 # The right granted on the created socket chmod-socket = 666 # Plugin to use and interpreter config single-interpreter = true master = true plugin = python3 lazy-apps = true enable-threads = 4 # Module to import module = searx.webapp # Virtualenv and python path pythonpath = /usr/local/searxng/ chdir = /usr/local/searxng/searx/ # automatically set processes name to something meaningful auto-procname = true # Disable request logging for privacy disable-logging = true log-5xx = true # Set the max size of a request (request-body excluded) buffer-size = 8192 # No keep alive # See https://github.com/searx/searx-docker/issues/24 add-header = Connection: close # uwsgi serves the static files static-map = /static=/usr/local/searxng/searx/static # expires set to one day static-expires = /* 86400 static-gzip-all = True offload-threads = 4 ================================================ FILE: server.py ================================================ # -*- coding: utf-8 -*- import time import uvicorn import sys import getopt import json import os from pprint import pprint import requests import trafilatura from trafilatura import bare_extraction from concurrent.futures import ThreadPoolExecutor import concurrent import requests import openai import time from datetime import datetime from urllib.parse import urlparse import platform import urllib.parse import free_ask_internet from pydantic import BaseModel, Field from fastapi import FastAPI, HTTPException from fastapi.middleware.cors import CORSMiddleware from contextlib import asynccontextmanager from typing import Any, Dict, List, Literal, Optional, Union from sse_starlette.sse import ServerSentEvent, EventSourceResponse from fastapi.responses import StreamingResponse app = FastAPI() app.add_middleware( CORSMiddleware, allow_origins=["*"], allow_credentials=True, allow_methods=["*"], allow_headers=["*"], ) class ModelCard(BaseModel): id: str object: str = "model" created: int = Field(default_factory=lambda: int(time.time())) owned_by: str = "owner" root: Optional[str] = None parent: Optional[str] = None permission: Optional[list] = None class ModelList(BaseModel): object: str = "list" data: List[ModelCard] = [] class ChatMessage(BaseModel): role: Literal["user", "assistant", "system"] content: str class DeltaMessage(BaseModel): role: Optional[Literal["user", "assistant", "system"]] = None content: Optional[str] = None class QueryRequest(BaseModel): query:str model: str ask_type: Literal["search", "llm"] llm_auth_token: Optional[str] = "CUSTOM" llm_base_url: Optional[str] = "" using_custom_llm:Optional[bool] = False lang:Optional[str] = "zh-CN" class ChatCompletionRequest(BaseModel): model: str messages: List[ChatMessage] temperature: Optional[float] = None top_p: Optional[float] = None max_length: Optional[int] = None stream: Optional[bool] = False class ChatCompletionResponseChoice(BaseModel): index: int message: ChatMessage finish_reason: Literal["stop", "length"] class ChatCompletionResponseStreamChoice(BaseModel): index: int delta: DeltaMessage finish_reason: Optional[Literal["stop", "length"]] class ChatCompletionResponse(BaseModel): model: str object: Literal["chat.completion", "chat.completion.chunk"] choices: List[Union[ChatCompletionResponseChoice, ChatCompletionResponseStreamChoice]] created: Optional[int] = Field(default_factory=lambda: int(time.time())) class SearchItem(BaseModel): url: str icon_url: str site_name:str snippet:str title:str class SearchItemList(BaseModel): search_items: List[SearchItem] = [] class SearchResp(BaseModel): code:int msg:str data: List[SearchItem] = [] @app.get("/v1/models", response_model=ModelList) async def list_models(): global model_args model_card = ModelCard(id="gpt-3.5-turbo") return ModelList(data=[model_card]) @app.post("/v1/chat/completions", response_model=ChatCompletionResponse) async def create_chat_completion(request: ChatCompletionRequest): global model, tokenizer print(request) if request.messages[-1].role != "user": raise HTTPException(status_code=400, detail="Invalid request") query = request.messages[-1].content generate = predict(query, "", request.model) return EventSourceResponse(generate, media_type="text/event-stream") def predict(query: str, history: None, model_id: str): choice_data = ChatCompletionResponseStreamChoice( index=0, delta=DeltaMessage(role="assistant"), finish_reason=None ) chunk = ChatCompletionResponse(model=model_id, choices=[ choice_data], object="chat.completion.chunk") yield "{}".format(chunk.json(exclude_unset=True)) new_response = "" current_length = 0 for token in free_ask_internet.ask_internet(query=query): new_response += token if len(new_response) == current_length: continue new_text = new_response[current_length:] current_length = len(new_response) choice_data = ChatCompletionResponseStreamChoice( index=0, delta=DeltaMessage(content=new_text,role="assistant"), finish_reason=None ) chunk = ChatCompletionResponse(model=model_id, choices=[ choice_data], object="chat.completion.chunk") yield "{}".format(chunk.json(exclude_unset=True)) choice_data = ChatCompletionResponseStreamChoice( index=0, delta=DeltaMessage(), finish_reason="stop" ) chunk = ChatCompletionResponse(model=model_id, choices=[ choice_data], object="chat.completion.chunk") yield "{}".format(chunk.json(exclude_unset=True)) yield '[DONE]' @app.post("/api/search/get_search_refs", response_model=SearchResp) async def get_search_refs(request: QueryRequest): global search_results search_results = [] search_item_list = [] if request.ask_type == "search": search_links,search_results = free_ask_internet.search_web_ref(request.query) for search_item in search_links: snippet = search_item.get("snippet") url = search_item.get("url") icon_url = search_item.get("icon_url") site_name = search_item.get("site_name") title = search_item.get("title") si = SearchItem(snippet=snippet,url=url,icon_url=icon_url,site_name=site_name,title=title) search_item_list.append(si) resp = SearchResp(code=0,msg="success",data=search_item_list) return resp def generator(prompt:str, model:str, llm_auth_token:str,llm_base_url:str, using_custom_llm=False,is_failed=False): if is_failed: yield "搜索失败，没有返回结果" else: total_token = "" for token in free_ask_internet.chat(prompt=prompt,model=model,llm_auth_token=llm_auth_token,llm_base_url=llm_base_url,using_custom_llm=using_custom_llm,stream=True): total_token += token yield token @app.post("/api/search/stream/{search_uuid}") async def stream(search_uuid:str,request: QueryRequest): global search_results if request.ask_type == "llm": answer_language = ' Simplified Chinese ' if request.lang == "zh-CN": answer_language = ' Simplified Chinese ' if request.lang == "zh-TW": answer_language = ' Traditional Chinese ' if request.lang == "en-US": answer_language = ' English ' prompt = ' You are a large language AI assistant develop by nash_su. Answer user question in ' + answer_language + '. And here is the user question: ' + request.query generate = generator(prompt,model=request.model,llm_auth_token=request.llm_auth_token, llm_base_url=request.llm_base_url, using_custom_llm=request.using_custom_llm) else: prompt = None limit_count = 10 while limit_count > 0: try: if len(search_results) > 0: prompt = free_ask_internet.gen_prompt(request.query,search_results,lang=request.lang,context_length_limit=8000) break else: limit_count -= 1 time.sleep(1) except Exception as err: limit_count -= 1 time.sleep(1) total_token = "" if prompt: generate = generator(prompt,model=request.model,llm_auth_token=request.llm_auth_token, llm_base_url=request.llm_base_url, using_custom_llm=request.using_custom_llm) else: generate = generator(prompt,model=request.model,llm_auth_token=request.llm_auth_token,llm_base_url=request.llm_base_url, using_custom_llm=request.using_custom_llm,is_failed=True) # return EventSourceResponse(generate, media_type="text/event-stream") return StreamingResponse(generate, media_type="text/event-stream") def main(): port = 8000 search_results = [] uvicorn.run(app, host='0.0.0.0', port=port, workers=1) if __name__ == "__main__": main()