[ { "path": ".editorconfig", "content": "root = true\n\n[*]\ninsert_final_newline = true\n\n[*.json]\nindent_style = space\nindent_size = 4\n\n[*.schema]\nindent_style = space\nindent_size = 4\n" }, { "path": ".gitattributes", "content": "###############################################################################\r\n# Set default behavior to automatically normalize line endings.\r\n###############################################################################\r\n* text=crlf\r\n\r\n###############################################################################\r\n# Set default behavior for command prompt diff.\r\n#\r\n# This is need for earlier builds of msysgit that does not have it on by\r\n# default for csharp files.\r\n# Note: This is only used by command line\r\n###############################################################################\r\n#*.cs diff=csharp\r\n\r\n###############################################################################\r\n# Set the merge driver for project and solution files\r\n#\r\n# Merging from the command prompt will add diff markers to the files if there\r\n# are conflicts (Merging from VS is not affected by the settings below, in VS\r\n# the diff markers are never inserted). Diff markers may cause the following \r\n# file extensions to fail to load in VS. An alternative would be to treat\r\n# these files as binary and thus will always conflict and require user\r\n# intervention with every merge. To do so, just uncomment the entries below\r\n###############################################################################\r\n#*.sln merge=binary\r\n#*.csproj merge=binary\r\n#*.vbproj merge=binary\r\n#*.vcxproj merge=binary\r\n#*.vcproj merge=binary\r\n#*.dbproj merge=binary\r\n#*.fsproj merge=binary\r\n#*.lsproj merge=binary\r\n#*.wixproj merge=binary\r\n#*.modelproj merge=binary\r\n#*.sqlproj merge=binary\r\n#*.wwaproj merge=binary\r\n\r\n###############################################################################\r\n# behavior for image files\r\n#\r\n# image files are treated as binary by default.\r\n###############################################################################\r\n#*.jpg binary\r\n#*.png binary\r\n#*.gif binary\r\n\r\n###############################################################################\r\n# diff behavior for common document formats\r\n# \r\n# Convert binary document formats to text before diffing them. This feature\r\n# is only available from the command line. Turn it on by uncommenting the \r\n# entries below.\r\n###############################################################################\r\n#*.doc diff=astextplain\r\n#*.DOC diff=astextplain\r\n#*.docx diff=astextplain\r\n#*.DOCX diff=astextplain\r\n#*.dot diff=astextplain\r\n#*.DOT diff=astextplain\r\n#*.pdf diff=astextplain\r\n#*.PDF diff=astextplain\r\n#*.rtf diff=astextplain\r\n#*.RTF diff=astextplain\r\n" }, { "path": ".github/CODEOWNERS", "content": "# Lines starting with '#' are comments.\n# Each line is a file pattern followed by one or more owners.\n\n# More details are here: https://help.github.com/articles/about-codeowners/\n\n# The '*' pattern is global owners.\n\n# Order is important. The last matching pattern has the most precedence.\n# The folders are ordered as follows:\n\n# In each subsection folders are ordered first by depth, then alphabetically.\n# This should make it easy to add new rules without breaking existing ones.\n\n# Global rule:\n* @microsoft/botframework-sdk" }, { "path": ".github/ISSUE_TEMPLATE/bot-framework-bug.md", "content": "---\nname: Bug\nabout: Report a new Bot Framework bug.\ntitle: \"\"\nlabels: \"needs-triage, bug\"\nassignees: \"\"\n---\n\n### [Github issues](https://github.com/microsoft/botframework-sdk) should be used for bugs and feature requests. Use [Stack Overflow](https://stackoverflow.com/questions/tagged/botframework) for general \"how-to\" questions. \n\n## Version\nWhat package version of the SDK are you using.\n\n## Describe the bug\nGive a clear and concise description of what the bug is.\n\n## To Reproduce\nSteps to reproduce the behavior:\n1. Go to '...'\n2. Click on '....'\n3. Scroll down to '....'\n4. See error\n\n## Expected behavior\nGive a clear and concise description of what you expected to happen.\n\n## Screenshots\nIf applicable, add screenshots to help explain your problem.\n\n## Additional context\nAdd any other context about the problem here.\n\n## Tracking Status\n\n**Dotnet SDK** [TODO]()\n- [ ] PR\n- [ ] Merged\n\n**Javascript SDK** [TODO]()\n- [ ] PR\n- [ ] Merged\n\n**Python SDK** [TODO]()\n- [ ] PR\n- [ ] Merged\n\n**Java SDK** [TODO]()\n- [ ] PR\n- [ ] Merged\n\n**Samples** [TODO]()\n- [ ] PR\n- [ ] Merged\n\n**Docs** [TODO]()\n- [ ] PR\n- [ ] Merged\n\n**Tools** [TODO]()\n- [ ] PR\n- [ ] Merged" }, { "path": ".github/ISSUE_TEMPLATE/bot-framework-feature-request.md", "content": "---\nname: Feature Request\nabout: Create a new Bot Framework feature request.\ntitle: \"\"\nlabels: \"needs-triage, feature-request\"\nassignees: \"\"\n---\n\n## This is not for asking questions or filing bugs\n\nTo get help with an issue with your bot\n* [\"How To\"](https://stackoverflow.com/help/how-to-ask) questions should be submitted on [Stack Overflow](https://stackoverflow.com/questions/tagged/botframework).\n\nTo file an issue against a component please go to the components repo\n* [Create Azure Bot Framework Service or Channel issue](https://github.com/Microsoft/botframework-services/issues/new/choose)\n* [Create Bot Builder Tools issue](https://github.com/Microsoft/botbuilder-tools/issues/new/choose)\n* [Create Bot Builder SDK V3 (.NET or Javascript) issue](https://github.com/Microsoft/botbuilder-V3/issues/new/choose)\n* [Create Bot Builder SDK V4 .NET issue](https://github.com/Microsoft/botbuilder-dotnet/issues/new/choose)\n* [Create Bot Builder SDK V4 Javascript issue](https://github.com/Microsoft/botbuilder-js/issues/new/choose)\n* [Create Bot Builder SDK V4 Java issue](https://github.com/Microsoft/botbuilder-java/issues/new/choose)\n* [Create Bot Builder SDK V4 Python issue](https://github.com/Microsoft/botbuilder-python/issues/new/choose)\n\n## Issue\n\n*Describe the issue you are addressing*\n\n## Proposed change\n\n*Describe the proposed solution*\n\n## Component Impact \n\n*Describe which components need to be updated*\n\n## Customer Impact \n\n*Describe the impact on customers*\n\n## Tracking Status\n\n**Dotnet SDK** [TODO]()\n- [ ] PR\n- [ ] Merged\n\n**Javascript SDK** [TODO]()\n- [ ] PR\n- [ ] Merged\n\n**Python SDK** [TODO]()\n- [ ] PR\n- [ ] Merged\n\n**Java SDK** [TODO]()\n- [ ] PR\n- [ ] Merged\n\n**Samples** [TODO]()\n- [ ] PR\n- [ ] Merged\n\n**Docs** [TODO]()\n- [ ] PR\n- [ ] Merged\n\n**Tools** [TODO]()\n- [ ] PR\n- [ ] Merged\n" }, { "path": ".github/PULL_REQUEST_TEMPLATE.md", "content": "Fixes #\n\n## Proposed Changes\n\n\n\n - \n -\n -\n\n\n## Testing\n" }, { "path": ".github/issue_template.md", "content": "## This is not for asking questions or filing bugs\n\nTo get help with an issue with your bot\n* [\"How To\"](https://stackoverflow.com/help/how-to-ask) questions should be submitted on [Stack Overflow](https://stackoverflow.com/questions/tagged/botframework) where there is a robust community to help you\n\nTo file an issue against a component please go to the components repo\n* [Create Azure Bot Framework Service or Channel issue](https://github.com/Microsoft/botframework-services/issues/new/choose)\n* [Create Bot Builder Tools issue](https://github.com/Microsoft/botbuilder-tools/issues/new/choose)\n* [Create Bot Builder SDK V3 (.NET or Javascript) issue](https://github.com/Microsoft/botbuilder-V3/issues/new/choose)\n* [Create Bot Builder SDK V4 .NET issue](https://github.com/Microsoft/botbuilder-dotnet/issues/new/choose)\n* [Create Bot Builder SDK V4 Javascript issue](https://github.com/Microsoft/botbuilder-js/issues/new/choose)\n* [Create Bot Builder SDK V4 Java issue](https://github.com/Microsoft/botbuilder-java/issues/new/choose)\n* [Create Bot Builder SDK V4 Python issue](https://github.com/Microsoft/botbuilder-python/issues/new/choose)\n\n" }, { "path": ".gitignore", "content": "## Ignore Visual Studio temporary files, build results, and\n## files generated by popular Visual Studio add-ons.\n\n# User-specific files\n*.suo\n*.user\n*.userosscache\n*.sln.docstates\n\n# User-specific files (MonoDevelop/Xamarin Studio)\n*.userprefs\n\n# Build results\n[Dd]ebug/\n[Dd]ebugPublic/\n[Rr]elease/\n[Rr]eleases/\nx64/\nx86/\nbld/\n[Bb]in/\n[Oo]bj/\necf/\nrcf/\n\nbones/\n.vs/\n.vscode/\n.config/\n\n# Python virtual environments\n**/*env/\n" }, { "path": "CODE_OF_CONDUCT.md", "content": "# Microsoft Open Source Code of Conduct\n\nThis project has adopted the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/).\n\nResources:\n\n- [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/)\n- [Microsoft Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/)\n- Contact [opencode@microsoft.com](mailto:opencode@microsoft.com) with questions or concerns\n" }, { "path": "Common/Telemetry/README.md", "content": "\n\n# Telemetry Analytics\n\nBot Analytics provides telemetry for analyzing conversation activities.\n\n## Requirements\n\nTo enable bot analytics: \n\n* [Add Telemetry](https://aka.ms/AddBotTelemetry) via Application Insights in the Bot Code\n* [Enable Application Insights](https://docs.microsoft.com/en-us/azure/bot-service/bot-service-resources-app-insights-keys?view=azure-bot-service-4.0) on Azure Bot Service\n\nOnce active, basic visualization is available out of the box on Azure Bot Service. In addition, one can deploy and customize more advanced analytics views. \n\n## Conversation Health Dashboard\n\nThe Conversation Health Dashboard template presents analytics view of bot conversation activity. \n\n\n\n### Instructions\n\n\n\n\n\n## See Also\n\n* [Analyze Bot Telemetry](https://aka.ms/AnalyzeBotTelemetry) : Learn how to analyze Application Insights log analytics data.\n* See [Azure ARM templates](https://docs.microsoft.com/en-us/azure/azure-resource-manager/templates/quickstart-create-templates-use-the-portal) Documentation\n\n" }, { "path": "Common/Transcripts/CoreExtensionsTests/ConversationStateTest.chat", "content": "bot=bot\nuser=user\n\nuser:start test\nbot:bot message\nuser:set foo\nuser:read\nbot:value:foo\nuser:delete\nuser:read\nbot:value:\nuser:set bar\nuser:set test\nuser:read\nbot:value:test\nuser:read\nbot:value:test\nuser:delete\nuser:read\nbot:value:" }, { "path": "Common/Transcripts/CoreExtensionsTests/CustomStateTest.chat", "content": "bot=bot\nuser=user\n\nuser:start test\nbot:bot message\nuser:set foo\nuser:read\nbot:value:foo\nuser:delete\nuser:read\nbot:value:\nuser:set bar\nuser:set test\nuser:read\nbot:value:test\nuser:read\nbot:value:test\nuser:delete\nuser:read\nbot:value:" }, { "path": "Common/Transcripts/CoreExtensionsTests/UserStateTest.chat", "content": "bot=bot\nuser=user\n\nuser:start test\nbot:bot message\nuser:set foo\nuser:read\nbot:value:foo\nuser:delete\nuser:read\nbot:value:\nuser:set bar\nuser:set test\nuser:read\nbot:value:test\nuser:read\nbot:value:test\nuser:delete\nuser:read\nbot:value:" }, { "path": "Common/Transcripts/CoreTests/BotAdapted_Bracketing.chat", "content": "bot=bot\nuser=user\n\nbot:before message\nbot:conversationUpdate\nbot:after message\nuser:use middleware\nbot:before message\nbot:using middleware\nbot:after message\nuser:catch exception\nbot:before message\nbot:generating exception\nbot:Caught: exception to catch\n\n" }, { "path": "Common/Transcripts/DialogsTests/AttachmentPrompt.chat", "content": "user=User\nbot=Bot\n\nUser: hello\nBot: please add an attachment.\nUser: [Attachment=ContentAttachment.json]\nBot: some content" }, { "path": "Common/Transcripts/DialogsTests/ChoicePrompt.chat", "content": "user=user\nbot=bot\n\nuser: Hello!\nbot: favorite color? (1) red, (2) green, or (3) blue\nuser: red\nbot: Bot received the choice 'red'.\n\nuser: Hello!\nbot: favorite color? (1) red, (2) green, or (3) blue\nuser: 3\nbot: Bot received the choice 'blue'.\n\nuser: Hello!\nbot: favorite color? (1) red, (2) green, or (3) blue\nuser: cat\nbot: I didn't catch that. Select a color from the list. (1) red, (2) green, or (3) blue\nuser: green\nbot: Bot received the choice 'green'." }, { "path": "Common/Transcripts/DialogsTests/ConfirmPrompt.chat", "content": "user=User\nbot=Bot\n\nUser: hello\nBot: Please confirm.\nUser: hello\nBot: Please confirm, say 'yes' or 'no' or something like that.\nUser: no\nBot: Not confirmed.\n\nUser: hello\nBot: Please confirm.\nUser: yes\nBot: Confirmed.\n" }, { "path": "Common/Transcripts/DialogsTests/ContentAttachment.json", "content": "\"some content\"" }, { "path": "Common/Transcripts/DialogsTests/DateTimePrompt.chat", "content": "user=User\nbot=Bot\n\nUser: hello\nBot: What date would you like?\nUser: cat\nBot: Sorry, but that is not a date. What date would you like?\nUser: 31 February 2018\nBot: Timex:'2018-02-31' Value:'not resolved'\n\nUser: hello\nBot: What date would you like?\nUser: 5th December 2018 at 9am\nBot: Timex:'2018-12-05T09' Value:'2018-12-05 09:00:00'" }, { "path": "Common/Transcripts/DialogsTests/NumberPrompt.chat", "content": "user=User\nbot=Bot\n\nUser: hello\nBot: Enter a number.\nUser: hello\nBot: You must enter a valid positive number less than 100.\nUser: 150\nBot: You must enter a valid positive number less than 100.\nUser: 64\nBot: Bot received the number '64'.\n\nUser: hello\nBot: Enter a number.\nUser: One\nBot: Bot received the number '1'." }, { "path": "Common/Transcripts/DialogsTests/TextPrompt.chat", "content": "user=User\nbot=Bot\n\nUser: hello\nBot: Enter some text.\nUser: hi\nBot: Make sure the text is greater than three characters.\nUser: hello\nBot: Bot received the text 'hello'." }, { "path": "Common/Transcripts/DialogsTests/Waterfall.chat", "content": "user=User\nbot=Bot\n\nUser: hello\nBot: step1\nUser: hello\nBot: step2\nUser: hello\nBot: step3" }, { "path": "Common/Transcripts/DialogsTests/WaterfallNested.chat", "content": "user=User\nbot=Bot\n\nUser: hello\nBot: step1\nBot: step1.1\nUser: hello\nBot: step1.2\nUser: hello\nBot: step2\nBot: step2.1\nUser: hello\nBot: step2.2" }, { "path": "Common/Transcripts/DialogsTests/WaterfallPrompt.chat", "content": "user=User\nbot=Bot\n\nUser: hello\nBot: step1\nBot: Enter a number.\nUser: hello again\nBot: It must be a number\nUser: 42\nBot: Thanks for '42'\nBot: step2\nBot: Enter a number.\nUser: apple\nBot: It must be a number\nUser: orange\nBot: It must be a number\nUser: 64\nBot: Thanks for '64'\nBot: step3" }, { "path": "Common/Transcripts/LuisTests/LuisMiddleware.chat", "content": "user=User\nbot=Bot\n\nuser:test\nbot:default message\nuser:add imax theater to my upcoming events\nbot:intent:Calendar_Add\nuser:display weekend plans\nbot:intent:Calendar_Find\nuser:test\nbot:default message" }, { "path": "Common/Transcripts/QnATests/QnAMiddleware.chat", "content": "user=User\nbot=Bot\n\nuser:random text\nbot:default message\nuser:hi\nbot:hello\nuser:test\nbot:message\nuser:tests\nbot:message\nuser:custom random text\nbot:default message\n" }, { "path": "Common/Transcripts/README.md", "content": "# Bot Builder SDK v4 - Transcript Files\n\nThis folder contains the transcript files (.chat files) used in BotBuilder-v4 for testing several components and guarantee the same behavior between the different platform (.NET, NodeJS, Python, Java).\n\n## BotBuilder Developer writing new transcript-based tests\n\n1. Clone/Fork the [BotBuilder repository](https://github.com/Microsoft/BotBuilder).\n\n2. Create a branch and place the new transcripts in the [Common\\Transcripts folder](https://github.com/southworkscom/BotBuilder/tree/botbuilder-v4-transcripts/Common/Transcripts) (this can be done locally without actually pushing changes at this stage)\n\n3. Write code and create new tests. Set the `BOTBUILDER_TRANSCRIPTS_LOCATION` Environment var pointing to the local repo / transcript folder.\n\n E.g.:\n\n ```shell\n # On *nix:\n export BOTBUILDER_TRANSCRIPTS_LOCATION=~/projects/BotBuilder/Common/Transcripts\n ```\n\n\n ```Batchfile\n REM On Windows:\n SET BOTBUILDER_TRANSCRIPTS_LOCATION=C:\\Projects\\BotBuilder\\Common\\Transcripts\n ```\n\n4. Run the transcript tests from the platform repository. The code will look into the BOTBUILDER_TRANSCRIPTS_LOCATION folder for all files.\n\n E.g.:\n\n ```\n cd transcripts\n npm run test\n ```\n\n## Pushing changes to BotBuilder Repos\n\n1. Create a PR to the [BotBuilder repository](https://github.com/Microsoft/BotBuilder) with the new transcripts.\n\n2. Once the transcripts PR is merged, create a PR to the BotBuilder's platform repository with the code updates and tests.\n" }, { "path": "Common/Transcripts/TranslationTests/LocaleConvertToEnglish.chat", "content": "user=User\nbot=Bot\n\nuser:set language fr-ca\nuser:Set a meeting on 2018-10-04\nbot:message: Set a meeting on 10/4/2018\nuser:set language fr-fr\nuser:Set a meeting on 26/08/2018\nbot:message: Set a meeting on 8/26/2018\nuser:set language en-us\nuser:Set a meeting on 05/29/2018\nbot:message: Set a meeting on 05/29/2018\n" }, { "path": "Common/Transcripts/TranslationTests/TranslateToEnglish.chat", "content": "user=User\nbot=Bot\n\nuser:set language es-es\nuser:Hola\nbot:message: Hello\nuser:set language fr-fr\nuser:Salut\nbot:message: Hello\nuser:set language en-us\nuser:Hello\nbot:message: Hello\n" }, { "path": "Common/Transcripts/TranslationTests/TranslateToUserLanguage.chat", "content": "user=User\nbot=Bot\n\nuser:set language es-es\nuser:Hola\nbot:mensaje: Hola\nuser:set language fr-fr\nuser:Salut\nbot:message: Bonjour\nuser:set language en-us\nuser:Hello\nbot:message: Hello\n" }, { "path": "Common/badges/BotBuilderBadge.md", "content": "[![BotBuilder](\nhttps://img.shields.io/badge/Powered%20By-BotBuilder-blue.svg?colorA=black&label=Powered%20By&logo=data%3Aimage%2Fsvg%2Bxml%3Bbase64%2CPD94bWwgdmVyc2lvbj0iMS4wIiBlbmNvZGluZz0idXRmLTgiPz48c3ZnIGlkPSJtYXN0ZXItYXJ0Ym9hcmQiIHZpZXdCb3g9IjAgMCAxNDAwIDk4MCIgdmVyc2lvbj0iMS4xIiB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHg9IjBweCIgeT0iMHB4IiBzdHlsZT0iZW5hYmxlLWJhY2tncm91bmQ6bmV3IDAgMCAxNDAwIDk4MDsiIHdpZHRoPSIxNDAwcHgiIGhlaWdodD0iOTgwcHgiPjxnIHRyYW5zZm9ybT0ibWF0cml4KDYxLjI1MDAwMzgxNDY5NzI2NiwgMCwgMCwgNjEuMjUwMDAzODE0Njk3MjY2LCAyMDkuOTk5OTY4NDY0NjY0LCAtMC4wMDAwMDEwMTc3NTc5OTUyMTQxOTY5KSI%2BPHRpdGxlPmljX2JvdF9mcmFtZXdvcms8L3RpdGxlPjxnIGlkPSJMYXllcl8yIiBkYXRhLW5hbWU9IkxheWVyIDIiPjxnIGlkPSJMYXllcl8xLTIiIGRhdGEtbmFtZT0iTGF5ZXIgMSI%2BPHBhdGggZD0iTSA3LjI4IDggQSAwLjczIDAuNzMgMCAwIDEgNi41NSA4LjczIEEgMC43MyAwLjczIDAgMCAxIDUuODIgOCBBIDAuNzMgMC43MyAwIDAgMSA2LjU1IDcuMjcgQSAwLjczIDAuNzMgMCAwIDEgNy4yOCA4IFoiIHN0eWxlPSJmaWxsOiByZ2IoMjU1LCAyNTUsIDI1NSk7Ii8%2BPHBhdGggZD0iTSAxMC4xOCA4IEEgMC43MyAwLjczIDAgMCAxIDkuNDUgOC43MyBBIDAuNzMgMC43MyAwIDAgMSA4LjcyIDggQSAwLjczIDAuNzMgMCAwIDEgOS40NSA3LjI3IEEgMC43MyAwLjczIDAgMCAxIDEwLjE4IDggWiIgc3R5bGU9ImZpbGw6IHJnYigyNTUsIDI1NSwgMjU1KTsiLz48cGF0aCBkPSJNIDYuOTcwMDAwMjY3MDI4ODA5IDExLjc3MDAwMDQ1Nzc2MzY3MiBMIDMuMjAwMDAwMDQ3NjgzNzE2IDggTCA2Ljk3MDAwMDI2NzAyODgwOSA0LjIzMDAwMDAxOTA3MzQ4NiBMIDYuMjg5OTk5OTYxODUzMDI3IDMuNTM5OTk5OTYxODUzMDI3MyBMIDEuODMwMDAwMDQyOTE1MzQ0MiA4IEwgNi4yODk5OTk5NjE4NTMwMjcgMTIuNDYwMDAwMDM4MTQ2OTczIEwgNi45NzAwMDAyNjcwMjg4MDkgMTEuNzcwMDAwNDU3NzYzNjcyIFoiIHN0eWxlPSJmaWxsOiByZ2IoMjU1LCAyNTUsIDI1NSk7Ii8%2BPHBhdGggZD0iTSA5LjAyOTk5OTczMjk3MTE5MSAxMS43NzAwMDA0NTc3NjM2NzIgTCA5LjcxMDAwMDAzODE0Njk3MyAxMi40NjAwMDAwMzgxNDY5NzMgTCAxNC4xNzAwMDAwNzYyOTM5NDUgOCBMIDkuNzEwMDAwMDM4MTQ2OTczIDMuNTM5OTk5OTYxODUzMDI3MyBMIDkuMDI5OTk5NzMyOTcxMTkxIDQuMjMwMDAwMDE5MDczNDg2IEwgMTIuODAwMDAwMTkwNzM0ODYzIDggTCA5LjAyOTk5OTczMjk3MTE5MSAxMS43NzAwMDA0NTc3NjM2NzIgWiIgc3R5bGU9ImZpbGw6IHJnYigyNTUsIDI1NSwgMjU1KTsiLz48cGF0aCBkPSJNOCwwYTgsOCwwLDEsMCw4LDhBOCw4LDAsMCwwLDgsMFpNOCwxNWE3LDcsMCwxLDEsNy03QTcsNywwLDAsMSw4LDE1WiIgc3R5bGU9ImZpbGw6IHJnYigyNTUsIDI1NSwgMjU1KTsiLz48L2c%2BPC9nPjwvZz48L3N2Zz4%3D)](https://github.com/Microsoft/BotBuilder)\n\nMarkdown\n```\n[![BotBuilder](\nhttps://img.shields.io/badge/Powered%20By-BotBuilder-blue.svg?colorA=black&label=Powered%20By&logo=data%3Aimage%2Fsvg%2Bxml%3Bbase64%2CPD94bWwgdmVyc2lvbj0iMS4wIiBlbmNvZGluZz0idXRmLTgiPz48c3ZnIGlkPSJtYXN0ZXItYXJ0Ym9hcmQiIHZpZXdCb3g9IjAgMCAxNDAwIDk4MCIgdmVyc2lvbj0iMS4xIiB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHg9IjBweCIgeT0iMHB4IiBzdHlsZT0iZW5hYmxlLWJhY2tncm91bmQ6bmV3IDAgMCAxNDAwIDk4MDsiIHdpZHRoPSIxNDAwcHgiIGhlaWdodD0iOTgwcHgiPjxnIHRyYW5zZm9ybT0ibWF0cml4KDYxLjI1MDAwMzgxNDY5NzI2NiwgMCwgMCwgNjEuMjUwMDAzODE0Njk3MjY2LCAyMDkuOTk5OTY4NDY0NjY0LCAtMC4wMDAwMDEwMTc3NTc5OTUyMTQxOTY5KSI%2BPHRpdGxlPmljX2JvdF9mcmFtZXdvcms8L3RpdGxlPjxnIGlkPSJMYXllcl8yIiBkYXRhLW5hbWU9IkxheWVyIDIiPjxnIGlkPSJMYXllcl8xLTIiIGRhdGEtbmFtZT0iTGF5ZXIgMSI%2BPHBhdGggZD0iTSA3LjI4IDggQSAwLjczIDAuNzMgMCAwIDEgNi41NSA4LjczIEEgMC43MyAwLjczIDAgMCAxIDUuODIgOCBBIDAuNzMgMC43MyAwIDAgMSA2LjU1IDcuMjcgQSAwLjczIDAuNzMgMCAwIDEgNy4yOCA4IFoiIHN0eWxlPSJmaWxsOiByZ2IoMjU1LCAyNTUsIDI1NSk7Ii8%2BPHBhdGggZD0iTSAxMC4xOCA4IEEgMC43MyAwLjczIDAgMCAxIDkuNDUgOC43MyBBIDAuNzMgMC43MyAwIDAgMSA4LjcyIDggQSAwLjczIDAuNzMgMCAwIDEgOS40NSA3LjI3IEEgMC43MyAwLjczIDAgMCAxIDEwLjE4IDggWiIgc3R5bGU9ImZpbGw6IHJnYigyNTUsIDI1NSwgMjU1KTsiLz48cGF0aCBkPSJNIDYuOTcwMDAwMjY3MDI4ODA5IDExLjc3MDAwMDQ1Nzc2MzY3MiBMIDMuMjAwMDAwMDQ3NjgzNzE2IDggTCA2Ljk3MDAwMDI2NzAyODgwOSA0LjIzMDAwMDAxOTA3MzQ4NiBMIDYuMjg5OTk5OTYxODUzMDI3IDMuNTM5OTk5OTYxODUzMDI3MyBMIDEuODMwMDAwMDQyOTE1MzQ0MiA4IEwgNi4yODk5OTk5NjE4NTMwMjcgMTIuNDYwMDAwMDM4MTQ2OTczIEwgNi45NzAwMDAyNjcwMjg4MDkgMTEuNzcwMDAwNDU3NzYzNjcyIFoiIHN0eWxlPSJmaWxsOiByZ2IoMjU1LCAyNTUsIDI1NSk7Ii8%2BPHBhdGggZD0iTSA5LjAyOTk5OTczMjk3MTE5MSAxMS43NzAwMDA0NTc3NjM2NzIgTCA5LjcxMDAwMDAzODE0Njk3MyAxMi40NjAwMDAwMzgxNDY5NzMgTCAxNC4xNzAwMDAwNzYyOTM5NDUgOCBMIDkuNzEwMDAwMDM4MTQ2OTczIDMuNTM5OTk5OTYxODUzMDI3MyBMIDkuMDI5OTk5NzMyOTcxMTkxIDQuMjMwMDAwMDE5MDczNDg2IEwgMTIuODAwMDAwMTkwNzM0ODYzIDggTCA5LjAyOTk5OTczMjk3MTE5MSAxMS43NzAwMDA0NTc3NjM2NzIgWiIgc3R5bGU9ImZpbGw6IHJnYigyNTUsIDI1NSwgMjU1KTsiLz48cGF0aCBkPSJNOCwwYTgsOCwwLDEsMCw4LDhBOCw4LDAsMCwwLDgsMFpNOCwxNWE3LDcsMCwxLDEsNy03QTcsNywwLDAsMSw4LDE1WiIgc3R5bGU9ImZpbGw6IHJnYigyNTUsIDI1NSwgMjU1KTsiLz48L2c%2BPC9nPjwvZz48L3N2Zz4%3D)](https://github.com/Microsoft/BotBuilder)\n```" }, { "path": "Contributing.md", "content": "# Instructions for Contributing Code\r\n\r\n## Contributing bug fixes and features\r\n\r\nThe Bot Framework team is currently accepting contributions in the form of bug fixes and new \r\nfeatures. Any submission must have an issue tracking it in the issue tracker that has\r\n been approved by the Bot Framework team. Your pull request should include a link to \r\n the bug that you are fixing. If you've submitted a PR for a bug, please post a \r\n comment in the bug to avoid duplication of effort.\r\n\r\n## Legal\r\n\r\nIf your contribution is more than 15 lines of code, you will need to complete a Contributor \r\nLicense Agreement (CLA). Briefly, this agreement testifies that you are granting us permission\r\n to use the submitted change according to the terms of the project's license, and that the work\r\n being submitted is under appropriate copyright.\r\n\r\nPlease submit a Contributor License Agreement (CLA) before submitting a pull request. \r\nYou may visit https://cla.azure.com to sign digitally. Alternatively, download the \r\nagreement ([Microsoft Contribution License Agreement.docx](https://www.codeplex.com/Download?ProjectName=typescript&DownloadId=822190) or\r\n [Microsoft Contribution License Agreement.pdf](https://www.codeplex.com/Download?ProjectName=typescript&DownloadId=921298)), sign, scan, \r\n and email it back to . Be sure to include your github user name along with the agreement. Once we have received the \r\n signed CLA, we'll review the request. \r\n" }, { "path": "LICENSE", "content": "The MIT License (MIT)\n\nCopyright (c) 2016 Microsoft\n\nPermission is hereby granted, free of charge, to any person obtaining a copy\nof this software and associated documentation files (the \"Software\"), to deal\nin the Software without restriction, including without limitation the rights\nto use, copy, modify, merge, publish, distribute, sublicense, and/or sell\ncopies of the Software, and to permit persons to whom the Software is\nfurnished to do so, subject to the following conditions:\n\nThe above copyright notice and this permission notice shall be included in all\ncopies or substantial portions of the Software.\n\nTHE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\nIMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\nFITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\nAUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\nLIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\nOUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\nSOFTWARE.\n" }, { "path": "Orchestrator/README.md", "content": "# Orchestrator\n\nConversational AI applications today are built using disparate technologies to fulfill language understanding (LU) needs, such as [LUIS][1] and [QnA Maker][2]. Often, conversational AI applications are also built by accessing subsequent [skills][3], each of which handle a specific conversation topic and can be built using different LU technologies. Hence, conversational AI applications typically require LU to route an incoming user request to an appropriate skill or to dispatch to a specific sub-component.\n\nOrchestrator is an LU solution optimized for conversational AI applications. It is built ground-up to run locally with your bot. See the [technical overview][18] for additional details.\n\n## Scenarios\n\n**Routing**: For bots, Orchestrator can replace the [LUIS Dispatch tool][5]. You can use Orchestrator instead of Dispatch to arbitrate between multiple [LUIS][1] and [QnA Maker][2] applications. With Orchestrator, you are likely to see:\n\n- Improved classification accuracy.\n- Higher resilience to data imbalance across your LUIS and QnA Maker authoring data.\n- Ability to correctly dispatch from relatively little authoring data.\n\n**Intent recognition**: You can use Orchestrator as an intent recognizer with [adaptive dialogs][6] to route user input to an appropriate skill or sub-component.\n\n**Entity extraction** is currently experimental and not yet for production use.\n\n## Authoring experience\n\nOrchestrator can be used in different development environments:\n\n- [Bot Framework SDK][24]: Orchestrator can be integrated into your code project by replacing LUIS for intent recognition, such as for skill delegation or dispatching to subsequent language understanding services. See the [SDK integration](#sdk-integration) section for more information. \n- [Bot Framework Composer][19]: Orchestrator can be selected as a recognizer within Bot Framework Composer. At this point there are limitations to using Orchestrator in Composer, primarily around importing of existing models and tuning recognition performance. (To use Orchestrator, enable the feature flag in your Composer settings.) See the [Composer integration](#composer-integration) section for more information.\n\nIn most cases, the [Bot Framework CLI][7] is required to prepare and optimize the model for your domain. The [BF Orchestrator command usage][23] page describes how to create, evaluate, and use an Orchestrator model. This diagram illustrates the first part of that process. \n\n

\n \n

\n\n**Note**: To use the CLI, first install the [Bot Framework CLI][7].\n\nSee the [BF Orchestrator command usage][23] page for instructions on how to create and optimize the language model for your bot.\n\n## SDK integration\n\nTo use Orchestrator in place of Dispatch in an existing bot:\n\n- Create an _Orchestrator recognizer_ and provide it the path to the base model and your snapshot.\n- Use the recognizer's _recognize_ method to recognize user input.\n\n### In a C\\# bot\n\n- Install the `Microsoft.Bot.Builder.AI.Orchestrator` NuGet package.\n- Install the latest supported version of the [Visual C++ redistributable package](https://support.microsoft.com/help/2977003/the-latest-supported-visual-c-downloads).\n\n```csharp\nusing Microsoft.Bot.Builder.AI.Orchestrator;\n\n// Get Model and Snapshot path.\nstring modelPath = Path.GetFullPath(OrchestratorConfig.ModelPath);\nstring snapshotPath = Path.GetFullPath(OrchestratorConfig.SnapshotPath);\n\n// Create OrchestratorRecognizer.\nOrchestratorRecognizer orc = new OrchestratorRecognizer()\n{\n ModelPath = modelPath,\n SnapshotPath = snapshotPath\n};\n\n// Recognize user input.\nvar recoResult = await orc.RecognizeAsync(turnContext, cancellationToken);\n```\n\n### In a JavaScript bot\n\n- Install the `botbuilder-ai-orchestrator` npm package to your bot.\n\n```javascript\nconst { OrchestratorRecognizer } = require('botbuilder-ai-orchestrator');\n\n// Create OrchestratorRecognizer.\nconst dispatchRecognizer = new OrchestratorRecognizer().configure({\n modelPath: process.env.ModelPath,\n snapshotPath: process.env.SnapShotPath\n});\n// To recognize user input\nconst recoResult = await dispatchRecognizer.recognize(context);\n```\n\n## Composer integration\n\nOrchestrator can be used as recognizer in [Bot Framework Composer][19]. \n\nIn general, to specify Orchestrator as a dialog recognizer:\n\n1. Select **Orchestrator** in the **Recognizer Type** drop-down menu for your bot.\n2. Review, evaluate and adjust examples in language data as you would normally for LUIS to ensure recognition quality. \n\nThis enables basic intent recognition. For more advanced scenarios follow the steps above to import and tune up routing quality. For more information about recognizers in Composer, see the discussion of [recognizers](https://docs.microsoft.com/composer/concept-dialog#recognizer) with respect to dialogs in Composer. Please make sure to follow the latest up-to-date instructions in [Composer documentation][25]\n\n### Model Versions \nComposer is using the pre-selected default base models. With the CLI one can download & use alternate models. To force Composer to use different models, one can specify in the bot advanced settings as follows.\n\nThe user can change the model by adding the following to appsettings.json:\n\n```\n\"orchestrator\": {\n \"model\": {\n \"en_intent\": \"pretrained.20200924.microsoft.dte.00.03.en.onnx\",\n \"multilingual_intent\": \"pretrained.20200924.microsoft.dte.00.03.multilingual.onnx\"\n }\n }\n ```\nIf this section is blank or unreadable, we simply use the Orchestrator defaults. If the model is not in the supported Orchestrator list, we throw an error message.\n\nSee more on models [here][20].\n\n## Limitations\n\n* Orchestrator is limited to intents only. Entity definitions are ignored and no entity extraction is performed during recognition.\n\n## Platform Support\nOrchestrator supports the following platforms.\n\n### OS Support\nMacOS v10.14 / v10.15\nUbuntu 18 / 20\nWindows 10\n\n### Language Support\nNodejs v10, v12, v14\nC# .NET Standard 2.1\nC# .NET Core 3.1\n\n\n## Additional Reading\n\n- [Tech overview][18]\n- [API reference][14]\n- [Roadmap](./docs/Overview.md#Roadmap)\n- [BF CLI Orchestrator commands][11]\n- [C# samples][12]\n- [NodeJS samples][13]\n- [BF Orchestrator Command Usage][23]\n\nSee the [School skill navigator](https://github.com/microsoft/BotBuilder-Samples/tree/main/composer-samples/csharp_dotnetcore/projects/OrchestratorSchoolNavigator) for an example of using Orchestrator commandlets to improve the quality of a .lu training set and using Composer to build a bot from examples in .lu format.\n\n[1]:https://luis.ai\n[2]:https://qnamaker.ai\n[3]:https://docs.microsoft.com/en-us/azure/bot-service/skills-conceptual\n[4]:https://en.wikipedia.org/wiki/Transformer_(machine_learning_model)\n[5]:https://docs.microsoft.com/azure/bot-service/bot-builder-tutorial-dispatch?tabs=cs\n[6]:https://aka.ms/adaptive-dialogs\n[7]:https://github.com/microsoft/botframework-cli\n[8]:https://github.com/microsoft/botframework-cli/tree/main/packages/luis#bf-luisversionexport\n[9]:https://github.com/microsoft/botframework-cli/tree/main/packages/luis#bf-luisconvert\n[10]:https://github.com/microsoft/botframework-cli/tree/main/packages/qnamaker#bf-qnamakerkbexport\n[11]:https://github.com/microsoft/botframework-cli/tree/main/packages/orchestrator\n[12]:https://github.com/microsoft/BotBuilder-Samples/tree/main/samples/csharp_dotnetcore/14.nlp-with-orchestrator\n[13]:https://github.com/microsoft/BotBuilder-Samples/tree/main/samples/javascript_nodejs/14.nlp-with-orchestrator\n[14]:https://aka.ms/bforchestratorapi\n[16]:https://github.com/microsoft/botframework-cli/tree/main/packages/orchestrator#bf-orchestratorcreate\n[18]:./docs/Overview.md\n[19]: https://docs.microsoft.com/composer/introduction\n[20]: https://aka.ms/NLRModels \"Natural Language Representation Models\"\n[21]:https://docs.microsoft.com/azure/bot-service/file-format/bot-builder-lu-file-format \"LU file format\"\n[22]:./docs/BFOrchestratorReport.md \"report interpretation\"\n[23]: ./docs/BFOrchestratorUsage.md \"BF Orchestrator command usage\"\n[24]:https://docs.microsoft.com/azure/bot-service/index-bf-sdk\n[25]: https://github.com/microsoft/BotFramework-Composer/blob/main/docs/preview%20features/orchestrator.md \"Composer integration instructions\"" }, { "path": "Orchestrator/Samples/CLI/ModelTuning/README.md", "content": "# Improve Language Model with BF CLI\n\nThe following sample illustrates how to evaluate and improve a simple language model using the Orchestrator report. You may adopt this process simply by editing the [demo.cmd](./demo.cmd) script to use your language model files.\n\n## Prerequisites\n\n* [Bot Framework CLI][5]\n* [Bot Framework CLI Orchestrator plugin][1] \n* An understanding of [Orchestrator][6] feature usage.\n\n## Walkthrough\n\nThe following files make up this illustration:\n\n```\ndemo.cmd: Script run evaluation and produces report\ncommon.lu: Bot Language Model LU\ncommon.test.lu: A test set containing examples not present in original LU\ncommon.fixed.lu: A corrected Language Model based on evaluation run\n```\n\nAssume a bot with a simple language model in [common.lu](./common.lu) file as follows:\n\n```\n# Help\n- help\n- i need help\n- please help\n- can you please help\n- what do you do\n- what is this bot for\n\n\n# BuySurface\n- How can I buy {ProductType=Surface PRO}\n- I want to buy {ProductType=Surface PRO}\n- I want to buy {ProductType=Surface laptop}\n- Can I buy {ProductType=Surface PRO} online\n\n# USConstitution\n- Tell me about the US Constitution\n- US Constitution\n- Info on the Constitution\n- tell me about the Constitution\n- constitution knowledgebase\n- constitution faq\n- what do you know about the constitution\n- The united states constitution\n\n# Restart\n- restart\n- reset\n- start over\n- menu\n- top menu\n\n# Finance\n- pay my bills\n- transfer funds\n- what's my balance\n- get $500 cash\n\n# LUIS\n- luis\n\n# Orchestrator\n- Orchestrator\n- Orch\n\n```\n\nThis evaluation examines how the Orchestrator engine can generalize language understanding from a few examples to include phrases never seen before. Hence, for proper language evaluation, the test file [common.test.lu](./common.test.lu) should contain utterances not present in the original \"training\" set:\n\n```\n# Help\n- help\n- do you have help\n- any assistance in this bot?\n\n# BuySurface\n- Looking for a computer made by MSFT\n- what kind of MS products do you have?\n\n# USConstitution\n- Where can I read our founding fathers famous document?\n- Is there an analysis of the bill of rights?\n- What is that proclamation thing?\n\n# Restart\n- reset\n- go to the beginning\n\n# Finance\n- can i pay with credit card?\n- do you use Zelle?\n- what is my bank borrowing limit?\n\n# LUIS\n- Language Understanding Inteligent Service\n\n# Orchestrator\n- Conductor\n\n```\n\nNext, run the [demo.cmd](./demo.cmd) script. \n\nIn first run you need to download the Orchestrator basemodel so execute ```demo.cmd getmodel``` which will download the model and run the test. Consequent runs can reuse the same basemodel (hint: see ```bf orchestrator:basemodel:list``` for alternate models). Also, if you wish to compare to [LUIS](https://luis.ai) results, edit the script with your LUIS application info.\n\nThis will produce a report in report folder such as follows:\n\n![Before Correction](./reportbefore.png)\n\nNotice how 5 utterances were misclassified. For example ```do you use Zelle?``` was classified as *Help* (with low score of 0.2998) instead of *Finance*. Also, note that what led to this misclassification is that the nearest example scored 0.5669 under the *Help* label. \n\nThe observation shows that the concepts for those misclassified utterances are not present in the original common.lu language model. To correct, we'll add representative examples in the original common.lu file. Note how we don't add the exact utterances, only representative examples:\n\n![Compared LU](./comparedLU.png)\n\nThe corrected file is available for reference as [common.fixed.lu](./common.fixed.lu). You may copy it over the common.lu.\n\nNow, re-run the test ```demo.cmd``` and view the resulting report:\n\n\n\n![Report After](./reportafter.png)\n\n## Summary\n\nThis walkthrough showed how to improve your bot's language model before ever deploying it. It uses BF CLI to test the language model, and use the resulting report to correct the language model. One should construct a language model with as many examples representing expected user utterances. However, it also illustrates how one does not need to account for all utterance permutations, rather only to present concepts within those utterances. The Orchestrator engine generalizes and can accurately detect similar utterances. \n\n## Additional Reading\n\n- [Orchestrator Documentation][6]\n- [BF CLI Orchestrator Command Usage][4]\n- [Report Interpretation][3]\n- [LU File Format][2]\n\n\n\n[1]: https://aka.ms/bforchestratorcli\t\"Orchestrator Plugin\"\n[2]: https://docs.microsoft.com/en-us/azure/bot-service/file-format/bot-builder-lu-file-format?view=azure-bot-service-4.0\t\"LU file format\"\n[3]: https://aka.ms/bforchestratorreport \"report interpretation\"\n[4]: https://github.com/microsoft/botframework-sdk/blob/main/Orchestrator/docs/BFOrchestratorUsage.md \"BF Orchestrator usage\"\n[5]: https://github.com/microsoft/botframework-cli\n[6]: https://aka.ms/bf-orchestrator\n\n\n\n\n\n\n\n\n\n\n\n\n\n" }, { "path": "Orchestrator/Samples/CLI/ModelTuning/common.fixed.lu", "content": "# Help\n- help\n- i need help\n- please help\n- can you please help\n- what do you do\n- what is this bot for\n\n\n# BuySurface\n- How can I buy {ProductType=Surface PRO}\n- I want to buy {ProductType=Surface PRO}\n- I want to buy {ProductType=Surface laptop}\n- Can I buy {ProductType=Surface PRO} online\n- Types of MS products?\n\n# USConstitution\n- Tell me about the US Constitution\n- US Constitution\n- Info on the Constitution\n- tell me about the Constitution\n- constitution knowledgebase\n- constitution faq\n- what do you know about the constitution\n- The united states constitution\n\n# Restart\n- restart\n- reset\n- start over\n- menu\n- top menu\n\n# Finance\n- pay my bills\n- transfer funds\n- what's my balance\n- get $500 cash\n- use Zelle\n- borrowing limits?\n\n# LUIS\n- luis\n- Language Understanding\n\n# Orchestrator\n- Orchestrator\n- Orch\n- Conductor organizer\n\n" }, { "path": "Orchestrator/Samples/CLI/ModelTuning/common.lu", "content": "# Help\n- help\n- i need help\n- please help\n- can you please help\n- what do you do\n- what is this bot for\n\n\n# BuySurface\n- How can I buy {ProductType=Surface PRO}\n- I want to buy {ProductType=Surface PRO}\n- I want to buy {ProductType=Surface laptop}\n- Can I buy {ProductType=Surface PRO} online\n\n# USConstitution\n- Tell me about the US Constitution\n- US Constitution\n- Info on the Constitution\n- tell me about the Constitution\n- constitution knowledgebase\n- constitution faq\n- what do you know about the constitution\n- The united states constitution\n\n# Restart\n- restart\n- reset\n- start over\n- menu\n- top menu\n\n# Finance\n- pay my bills\n- transfer funds\n- what's my balance\n- get $500 cash\n\n# LUIS\n- luis\n\n# Orchestrator\n- Orchestrator\n- Orch\n\n" }, { "path": "Orchestrator/Samples/CLI/ModelTuning/common.test.lu", "content": "# Help\n- help\n- do you have help\n- any assistance in this bot?\n\n\n# BuySurface\n- Looking for a computer made by MSFT\n- what kind of MS products do you have?\n\n# USConstitution\n- Where can I read our founding fathers famous document?\n- Is there an analysis of the bill of rights?\n- What is that proclamation thing?\n\n# Restart\n- reset\n- go to the beginning\n\n# Finance\n- can i pay with credit card?\n- do you use Zelle?\n- what is my bank borrowing limit?\n\n\n# LUIS\n- Language Understanding Inteligent Service\n\n# Orchestrator\n- Conductor\n\n" }, { "path": "Orchestrator/Samples/CLI/ModelTuning/demo.cmd", "content": "@echo off\n@echo ORCHESTRATOR EVALUATION DEMO\n\n@rem set SEED for different test sets\nset SEED=common\nset BLU=generated\\%SEED%.blu \nset LUFILE=%SEED%.lu\n@rem test file contains sample utterances that are not in main LU file.\nset TESTFILE=%SEED%.test.lu\n@rem proper test\n@rem set TESTFILE=%SEED%.test.lu\n\n\nset LUISKEY=\nset LUISAPP=\nset LUISHOST=\n\nset SKIPLUIS=0\nset QUERYRUN=0\n\nif \"%LUISKEY%\" == \"\" (\n @echo Skipping comparison with LUIS. Fill in LUIS info to compare results.\n set SKIPLUIS=1\n)\n\n@rem set QUERY to run a single utterance test\nset QUERY=\"what is the american declaration of independence?\"\n\nif \"%1\" == \"qonly\" (\n set QUERYRUN=1\n goto QUERYONLY\n)\n\n@rem model folder needs to be downloaded only once.\nif \"%1\" == \"getmodel\" (\n if EXIST model rd /s /q model\n)\n\n@echo cleaning folders\nif EXIST report (\nrd /s /q report && md report\n)\nif EXIST generated (\n rd /s /q generated && md generated\n)\n\n\n@rem Only need to retrieve model once\nIF NOT EXIST .\\model (\n@rem see bf orchestrator:basemodel:get --help to get the non-default model \n@rem see available models via bf orchestrator:basemodel:list\n @echo getting base model\n md model\n call bf orchestrator:basemodel:get --out model\n)\n@echo Create orchestrator snapshot .blu file\ncall bf orchestrator:create --model model --out %BLU% --in %LUFILE%\n\n@echo running orchestrator test to generate a report (see report folder) \ncall bf orchestrator:test --in %BLU% --model ./model --out report --test %TESTFILE%\n\nif \"%SKIPLUIS%\" == \"0\" (\n @echo running LUIS test...\n call bf luis:test --subscriptionKey %LUISKEY% --endpoint %LUISHOST% --appId %LUISAPP% --in %TESTFILE% --out report/luisresult.txt\n)\n\n:QUERYONLY\n@rem Illustrates how to query for only a single utterance. Edit %QUERY% above.\nif \"%QUERYRUN%\" == \"1\" (\n\n echo Orchestrator single utterance query:\n echo bf orchestrator:query --in %BLU% --model model --query %QUERY%\n call bf orchestrator:query --in %BLU% --model model --query %QUERY%\n\n\n if \"%SKIPLUIS%\" == \"0\" (\n echo LUIS single utterance query:\n echo bf luis:application:query --appId LUISAPP --endpoint LUISHOST --subscriptionKey LUISKEY --query %QUERY%\n call bf luis:application:query --appId %LUISAPP% --endpoint %LUISHOST% --subscriptionKey %LUISKEY% --query %QUERY%\n )\n)\n\n:DONE\n\n" }, { "path": "Orchestrator/Samples/dotnet/nlp-with-entities/.gitignore", "content": "# Assets generated by luis:build, luis:cross-train\n**/model\n**/[Gg]enerated\n**/*.blu\n*.json\n**/*.targets\n**/*.props\n**/*.cache\n*.dgspec.json" }, { "path": "Orchestrator/Samples/dotnet/nlp-with-entities/AdapterWithErrorHandler.cs", "content": "// Copyright (c) Microsoft Corporation. All rights reserved.\n// Licensed under the MIT License.\n\nusing System;\nusing Microsoft.Bot.Builder;\nusing Microsoft.Bot.Builder.Integration.AspNet.Core;\nusing Microsoft.Bot.Builder.TraceExtensions;\nusing Microsoft.Extensions.Configuration;\nusing Microsoft.Extensions.Logging;\n\nnamespace Microsoft.BotBuilderSamples\n{\n public class AdapterWithErrorHandler : BotFrameworkHttpAdapter\n {\n public AdapterWithErrorHandler(IConfiguration configuration, ILogger logger, ConversationState conversationState = null)\n : base(configuration, logger)\n {\n OnTurnError = async (turnContext, exception) =>\n {\n // Log any leaked exception from the application.\n logger.LogError(exception, $\"[OnTurnError] unhandled error : {exception.Message}\");\n\n // Send a message to the user\n await turnContext.SendActivityAsync(\"The bot encountered an error or bug.\");\n await turnContext.SendActivityAsync(\"To run this sample make sure you have the LUIS and QnA models deployed.\");\n await turnContext.SendActivityAsync(\"To continue to run this bot, please fix the bot source code.\");\n\n if (conversationState != null)\n {\n try\n {\n // Delete the conversationState for the current conversation to prevent the\n // bot from getting stuck in a error-loop caused by being in a bad state.\n // ConversationState should be thought of as similar to \"cookie-state\" in a Web pages.\n await conversationState.DeleteAsync(turnContext);\n }\n catch (Exception e)\n {\n logger.LogError(e, $\"Exception caught on attempting to Delete ConversationState : {e.Message}\");\n }\n }\n\n // Send a trace activity, which will be displayed in the Bot Framework Emulator\n await turnContext.TraceActivityAsync(\"OnTurnError Trace\", exception.Message, \"https://www.botframework.com/schemas/error\", \"TurnError\");\n };\n }\n }\n}\n" }, { "path": "Orchestrator/Samples/dotnet/nlp-with-entities/BotServices.cs", "content": "// Copyright (c) Microsoft Corporation. All rights reserved.\n// Licensed under the MIT License.\n\nusing Microsoft.Bot.Builder.AI.Luis;\nusing Microsoft.Bot.Builder.AI.Orchestrator;\nusing Microsoft.Bot.Builder.AI.QnA;\nusing Microsoft.Extensions.Configuration;\n\nnamespace Microsoft.BotBuilderSamples\n{\n public class BotServices : IBotServices\n {\n public BotServices(IConfiguration configuration, OrchestratorRecognizer dispatcher)\n {\n // Read the setting for cognitive services (LUIS, QnA) from the appsettings.json\n // If includeApiResults is set to true, the full response from the LUIS api (LuisResult)\n // will be made available in the properties collection of the RecognizerResult\n LuisHomeAutomationRecognizer = CreateLuisRecognizer(configuration, \"LuisHomeAutomationAppId\");\n LuisWeatherRecognizer = CreateLuisRecognizer(configuration, \"LuisWeatherAppId\");\n\n Dispatch = dispatcher;\n\n SampleQnA = new QnAMaker(new QnAMakerEndpoint\n {\n KnowledgeBaseId = configuration[\"QnAKnowledgebaseId\"],\n EndpointKey = configuration[\"QnAEndpointKey\"],\n Host = configuration[\"QnAEndpointHostName\"]\n });\n }\n\n public OrchestratorRecognizer Dispatch { get; private set; }\n \n public QnAMaker SampleQnA { get; private set; }\n \n public LuisRecognizer LuisHomeAutomationRecognizer { get; private set; }\n\n public LuisRecognizer LuisWeatherRecognizer { get; private set; }\n\n private LuisRecognizer CreateLuisRecognizer(IConfiguration configuration, string appIdKey)\n {\n var luisApplication = new LuisApplication(\n configuration[appIdKey],\n configuration[\"LuisAPIKey\"],\n configuration[\"LuisAPIHostName\"]);\n\n // Set the recognizer options depending on which endpoint version you want to use.\n // More details can be found in https://docs.microsoft.com/en-gb/azure/cognitive-services/luis/luis-migration-api-v3\n var recognizerOptions = new LuisRecognizerOptionsV2(luisApplication)\n {\n IncludeAPIResults = true,\n PredictionOptions = new LuisPredictionOptions()\n {\n IncludeAllIntents = true,\n IncludeInstanceData = true\n }\n };\n\n return new LuisRecognizer(recognizerOptions);\n }\n }\n}\n" }, { "path": "Orchestrator/Samples/dotnet/nlp-with-entities/Bots/DispatchBot.cs", "content": "// Copyright (c) Microsoft Corporation. All rights reserved.\n// Licensed under the MIT License.\n\nusing System.Collections.Generic;\nusing System.Linq;\nusing System.Threading;\nusing System.Threading.Tasks;\nusing Microsoft.Azure.CognitiveServices.Language.LUIS.Runtime.Models;\nusing Microsoft.Bot.Builder;\nusing Microsoft.Bot.Builder.Dialogs;\nusing Microsoft.Bot.Schema;\nusing Microsoft.Extensions.Logging;\n\nnamespace Microsoft.BotBuilderSamples\n{\n public class DispatchBot : ActivityHandler\n {\n private ILogger _logger;\n private IBotServices _botServices;\n\n public DispatchBot(IBotServices botServices, ILogger logger)\n {\n _logger = logger;\n _botServices = botServices;\n }\n\n protected override async Task OnMessageActivityAsync(ITurnContext turnContext, CancellationToken cancellationToken)\n {\n var dc = new DialogContext(new DialogSet(), turnContext, new DialogState());\n // Top intent tell us which cognitive service to use.\n var allScores = await _botServices.Dispatch.RecognizeAsync(dc, (Activity)turnContext.Activity, cancellationToken);\n var topIntent = allScores.Intents.First().Key;\n\n // Detected entities which could be used to help with intent dispatching, for example, when top intent score is too low or\n // when there are multiple top intents with close scores.\n var entities = allScores.Entities.Children().Where(t => t.Path != \"$instance\");\n \n // Next, we call the dispatcher with the top intent.\n await DispatchToTopIntentAsync(turnContext, topIntent, cancellationToken);\n }\n\n protected override async Task OnMembersAddedAsync(IList membersAdded, ITurnContext turnContext, CancellationToken cancellationToken)\n {\n const string WelcomeText = \"Type a greeting, or a question about the weather to get started.\";\n\n foreach (var member in membersAdded)\n {\n if (member.Id != turnContext.Activity.Recipient.Id)\n {\n await turnContext.SendActivityAsync(MessageFactory.Text($\"Welcome to Dispatch bot {member.Name}. {WelcomeText}\"), cancellationToken);\n }\n }\n }\n\n private async Task DispatchToTopIntentAsync(ITurnContext turnContext, string intent, CancellationToken cancellationToken)\n {\n switch (intent)\n {\n case \"HomeAutomation\":\n await ProcessHomeAutomationAsync(turnContext, cancellationToken);\n break;\n case \"Weather\":\n await ProcessWeatherAsync(turnContext, cancellationToken);\n break;\n case \"QnAMaker\":\n await ProcessSampleQnAAsync(turnContext, cancellationToken);\n break;\n default:\n _logger.LogInformation($\"Dispatch unrecognized intent: {intent}.\");\n await turnContext.SendActivityAsync(MessageFactory.Text($\"Dispatch unrecognized intent: {intent}.\"), cancellationToken);\n break;\n }\n }\n\n private async Task ProcessHomeAutomationAsync(ITurnContext turnContext, CancellationToken cancellationToken)\n {\n _logger.LogInformation(\"ProcessHomeAutomationAsync\");\n\n // Retrieve LUIS result for HomeAutomation.\n var recognizerResult = await _botServices.LuisHomeAutomationRecognizer.RecognizeAsync(turnContext, cancellationToken);\n var result = recognizerResult.Properties[\"luisResult\"] as LuisResult;\n\n var topIntent = result.TopScoringIntent.Intent; \n \n await turnContext.SendActivityAsync(MessageFactory.Text($\"HomeAutomation top intent {topIntent}.\"), cancellationToken);\n await turnContext.SendActivityAsync(MessageFactory.Text($\"HomeAutomation intents detected:\\n\\n{string.Join(\"\\n\\n\", result.Intents.Select(i => i.Intent))}\"), cancellationToken);\n if (result.Entities.Count > 0)\n {\n await turnContext.SendActivityAsync(MessageFactory.Text($\"HomeAutomation entities were found in the message:\\n\\n{string.Join(\"\\n\\n\", result.Entities.Select(i => i.Entity))}\"), cancellationToken);\n }\n }\n\n private async Task ProcessWeatherAsync(ITurnContext turnContext, CancellationToken cancellationToken)\n {\n _logger.LogInformation(\"ProcessWeatherAsync\");\n\n // Retrieve LUIS result for Weather.\n var recognizerResult = await _botServices.LuisWeatherRecognizer.RecognizeAsync(turnContext, cancellationToken);\n var result = recognizerResult.Properties[\"luisResult\"] as LuisResult;\n var topIntent = result.TopScoringIntent.Intent;\n\n await turnContext.SendActivityAsync(MessageFactory.Text($\"ProcessWeather top intent {topIntent}.\"), cancellationToken);\n await turnContext.SendActivityAsync(MessageFactory.Text($\"ProcessWeather Intents detected::\\n\\n{string.Join(\"\\n\\n\", result.Intents.Select(i => i.Intent))}\"), cancellationToken);\n if (result.Entities.Count > 0)\n {\n await turnContext.SendActivityAsync(MessageFactory.Text($\"ProcessWeather entities were found in the message:\\n\\n{string.Join(\"\\n\\n\", result.Entities.Select(i => i.Entity))}\"), cancellationToken);\n }\n }\n\n private async Task ProcessSampleQnAAsync(ITurnContext turnContext, CancellationToken cancellationToken)\n {\n _logger.LogInformation(\"ProcessSampleQnAAsync\");\n\n var results = await _botServices.SampleQnA.GetAnswersAsync(turnContext);\n if (results.Any())\n {\n await turnContext.SendActivityAsync(MessageFactory.Text(results.First().Answer), cancellationToken);\n }\n else\n {\n await turnContext.SendActivityAsync(MessageFactory.Text(\"Sorry, could not find an answer in the Q and A system.\"), cancellationToken);\n }\n }\n }\n}\n" }, { "path": "Orchestrator/Samples/dotnet/nlp-with-entities/CognitiveModels/HomeAutomation.lu", "content": "\n> LUIS application information\n> !# @app.name = Home Automation\n> !# @app.desc = Home Automation LUIS application - Bot Builder Samples\n> !# @app.versionId = 0.1\n> !# @app.culture = en-us\n> !# @app.luis_schema_version = 4.0.0\n> !# @app.tokenizerVersion = 1.0.0\n\n\n> # Intent definitions\n\n## ControlDevice\n- {@Room=breezeway} on please\n- change {@deviceProperty=temperature} to seventy two degrees\n- {@Device=coffee bar} on please\n- decrease {@deviceProperty=temperature} for me please\n- dim {@Room=kitchen} {@Device=lights} to 25 .\n- {@Device=fish pond} off please\n- {@Device=fish pond} on please\n- {@Device=illuminate} please\n- {@Room=living room} {@Device=lamp} on please\n- {@Room=living room} {@Device=lamps} off please\n- lock the {@Device=doors} for me please\n- lower your {@deviceProperty=volume}\n- make {@Device=camera 1} off please\n- make some {@Device=coffee}\n- play {@Device=dvd}\n- set {@Device=lights} out in {@Room=bedroom}\n- set {@Device=lights} to {@deviceProperty=bright}\n- set {@Device=lights} to {@deviceProperty=concentrate}\n- shut down my work {@Device=computer}\n- snap switch {@Device=fan} fifty percent\n- start {@Room=master bedroom} {@Device=light}.\n- {@Room=theater} on please\n- turn {@Device=dimmer} off\n- turn off {@Device=ac} please\n- turn off {@Room=foyer} {@Device=lights}\n- turn off {@Room=living room} {@Device=light}\n- turn off {@Device=staircase}\n- turn off venice {@Device=lamp}\n- turn on {@Room=bathroom} {@Device=heater}\n- turn on {@Device=external speaker}\n- turn on {@Room=kitchen} {@Device=faucet}\n- turn on {@Device=light} in {@Room=bedroom}\n- turn on my {@Room=bedroom} {@Device=lights}.\n- turn on the {@Room=furnace room} {@Device=lights}\n- turn on the internet in my {@Room=bedroom} please\n- turn on {@Device=thermostat} please\n- turn the {@Device=fan} to high\n- turn {@Device=thermostat} on 70.\n- {@Device_PatternAny} off [please]\n- {@Device_PatternAny} in {Room_PatternAny} on [please]\n- turn on {@Device_PatternAny} in {Room_PatternAny}\n- turn off {@Device_PatternAny} in {Room_PatternAny}\n- {@Device_PatternAny} in {Room_PatternAny} off [please]\n- {@Device_PatternAny} on [please]\n- turn on {@Device_PatternAny}\n\n\n## None\n\n\n> # Entity definitions\n\n@ ml Device\n\n@ ml deviceProperty\n\n@ ml Room\n\n\n> # PREBUILT Entity definitions\n\n@ prebuilt number\n\n\n> # Phrase list definitions\n\n\n> # List entities\n\n@ list Operation = \n\t- off :\n\t\t- turn off\n\t\t- switch off\n\t\t- lock\n\t\t- out\n\t\t- shut down\n\t\t- stop\n\t- on :\n\t\t- turn on\n\t\t- switch on\n\t\t- unlock\n\t\t- un lock\n\t\t- boot up\n\t\t- start\n\n\n> # RegEx entities\n\n\n> # Pattern.Any entities\n\n@ patternany Device_PatternAny\n@ patternany Room_PatternAny\n" }, { "path": "Orchestrator/Samples/dotnet/nlp-with-entities/CognitiveModels/QnAMaker.qna", "content": "# ? hi\n```\nhello\n```\n\n# ? greetings\n- good morning\n- good evening\n```\nHello!\n```\n\n# ? What are you?\n- What?\n- What do you do?\n- Who are you?\n- What is your name?\n- What should I call you?\n```\nI am the LUIS-QnAMaker Dispatch bot! This sample demonstrates using several LUIS applications and QnA Maker knowledge base using Orchestrator.\n```" }, { "path": "Orchestrator/Samples/dotnet/nlp-with-entities/CognitiveModels/Weather.lu", "content": "\n> LUIS application information\n> !# @app.name = Weather\n> !# @app.desc = Weather LUIS application - Bot Builder Samples\n> !# @app.versionId = 0.1\n> !# @app.culture = en-us\n> !# @app.luis_schema_version = 4.0.0\n\n\n> # Intent definitions\n\n## Get Weather Condition\n- current weather ?\n- do {@Location=florida} residents usually need ice scrapers\n- get {@Location=florence} temperature in september\n- get for me the weather conditions in {@Location=sonoma county}\n- get the daily temperature {@Location=greenwood indiana}\n- get the weather at {@Location=saint george utah}\n- how much rain does {@Location=chambersburg} get a year\n- show average rainfall for {@Location=boise}\n- temperature of {@Location=delhi} in celsius please\n- was last year about this time as wet as it is now in the {@Location=south} ?\n- what is the rain volume in {@Location=sonoma county} ?\n- what to wear in march in {@Location=california}\n- what's the weather like in {@Location=minneapolis}\n- weather in {@Location_PatternAny}\n- how's the weather in {@Location_PatternAny}\n- current weather in {@Location_PatternAny}\n\n\n## Get Weather Forecast\n- forecast in celcius\n- get the forcast for me\n- i want to know the temperature at {@Location=death valley}\n- provide me by {@Location=toronto} weather please\n- show me the forecast at {@Location=alabama}\n- soliciting today's weather\n- what is the weather in {@Location=redmond} ?\n- what is the weather today at 10 day {@Location=durham} ?\n- what will the weather be tomorrow in {@Location=new york} ?\n- what's the weather going to be like in {@Location=hawaii} ?\n- will it be raining in {@Location=ranchi}\n- will it rain this weekend\n- will it snow today\n- what's the forecast for next week in {@Location_PatternAny}\n- show me the forecast for {@Location_PatternAny}\n- what's the forecast for {@Location_PatternAny}\n\n\n## None\n\n\n> # Entity definitions\n\n@ ml Location\n\n\n> # PREBUILT Entity definitions\n\n\n> # Phrase list definitions\n\n\n> # List entities\n\n> # RegEx entities\n\n\n> # Pattern.Any entities\n\n@ patternany Location_PatternAny\n" }, { "path": "Orchestrator/Samples/dotnet/nlp-with-entities/Controllers/BotController.cs", "content": "// Copyright (c) Microsoft Corporation. All rights reserved.\n// Licensed under the MIT License.\n\nusing System.Threading.Tasks;\nusing Microsoft.AspNetCore.Mvc;\nusing Microsoft.Bot.Builder;\nusing Microsoft.Bot.Builder.Integration.AspNet.Core;\n\nnamespace Microsoft.BotBuilderSamples\n{\n [Route(\"api/messages\")]\n [ApiController]\n public class BotController : ControllerBase\n {\n private IBotFrameworkHttpAdapter _adapter;\n private IBot _bot;\n\n public BotController(IBotFrameworkHttpAdapter adapter, IBot bot)\n {\n _adapter = adapter;\n _bot = bot;\n }\n\n [HttpPost]\n public async Task PostAsync()\n {\n // Delegate the processingg of the HTTP POST to the adapter.\n // The adapter will invoke the bot.\n await _adapter.ProcessAsync(Request, Response, _bot);\n }\n }\n}\n" }, { "path": "Orchestrator/Samples/dotnet/nlp-with-entities/IBotServices.cs", "content": "// Copyright (c) Microsoft Corporation. All rights reserved.\n// Licensed under the MIT License.\n\nusing Microsoft.Bot.Builder.AI.Luis;\nusing Microsoft.Bot.Builder.AI.Orchestrator;\nusing Microsoft.Bot.Builder.AI.QnA;\n\nnamespace Microsoft.BotBuilderSamples\n{\n public interface IBotServices\n {\n LuisRecognizer LuisHomeAutomationRecognizer { get; }\n \n LuisRecognizer LuisWeatherRecognizer { get; }\n\n OrchestratorRecognizer Dispatch { get; }\n \n QnAMaker SampleQnA { get; }\n }\n}\n" }, { "path": "Orchestrator/Samples/dotnet/nlp-with-entities/OrchestratorConfig.cs", "content": "namespace Microsoft.BotBuilderSamples\n{\n public class OrchestratorConfig\n {\n public string SnapshotFile { get; set; }\n \n public string ModelFolder { get; set; }\n }\n}\n" }, { "path": "Orchestrator/Samples/dotnet/nlp-with-entities/Program.cs", "content": "// Copyright (c) Microsoft Corporation. All rights reserved.\n// Licensed under the MIT License.\n\nusing Microsoft.AspNetCore;\nusing Microsoft.AspNetCore.Hosting;\nusing Microsoft.Extensions.Logging;\n\nnamespace Microsoft.BotBuilderSamples\n{\n public class Program\n {\n public static void Main(string[] args)\n {\n CreateWebHostBuilder(args).Build().Run();\n }\n\n public static IWebHostBuilder CreateWebHostBuilder(string[] args) =>\n WebHost.CreateDefaultBuilder(args)\n .ConfigureLogging((logging) =>\n {\n logging.AddDebug();\n logging.AddConsole();\n })\n .UseStartup();\n }\n}\n" }, { "path": "Orchestrator/Samples/dotnet/nlp-with-entities/README.md", "content": "# NLP with Orchestrator\n\nBot Framework v4 NLP with Orchestrator (with entity recognition) bot sample\n\nThis bot has been created using [Bot Framework](https://dev.botframework.com), it shows how to create a bot that relies on multiple [LUIS.ai](https://www.luis.ai) and [QnAMaker.ai](https://www.qnamaker.ai) models for natural language processing (NLP).\n\nUse the Orchestrator dispatch model in cases when:\n\n- Your bot consists of multiple language modules (LUIS + QnA) and you need assistance in routing user's utterances to these modules in order to integrate the different modules into your bot.\n- Create a text classification model from text files.\n\n## Overview\n\nThis bot uses Orchestrator to route user utterances to multiple LUIS models and QnA maker services to support multiple conversational scenarios. This bot sample also demonstrates using Orchestrator for entity recognition.\n\n## Prerequisites\n\n| OS | Version | Architectures |\n| ------- | ------------------- | --------------- |\n| Windows | 10 (1607+) | ia32 (x86), x64 |\n| MacOS | 10.15+ | x64 |\n| Linux | Ubuntu 18.04, 20.04 | x64 |\n\nThis sample **requires** prerequisites in order to run.\n- Install latest supported version of [Visual C++ Redistributable](https://support.microsoft.com/en-gb/help/2977003/the-latest-supported-visual-c-downloads)\n\n- Install latest [Bot Framework Emulator](https://github.com/microsoft/BotFramework-Emulator/releases)\n\n- [.NET Core SDK](https://aka.ms/dotnet-core-applaunch?framework=Microsoft.AspNetCore.App&framework_version=3.1.0&arch=x64&rid=win10-x64) version 3.1\n \n ```bash\n > dotnet --version\n ```\n \n- Node v14\n \n ```bash\n > node -v \n ```\n \n- Install BF CLI with Orchestrator plugin\n \n ```bash\n > npm i -g @microsoft/botframework-cli\n ```\n Make sure bf orchestrator command is working and shows all available orchestrator commands\n ```bash\n > bf orchestrator\n ```\n\n## To try this bot sample\n\n- Clone the repository\n ```bash\n > git clone https://github.com/microsoft/botbuilder-samples.git\n ```\n \n- CD Orchestrator\\Samples\\dotnet\\nlp-with-entities\n ```bash\n > cd Orchestrator\\Samples\\dotnet\\nlp-with-entities\n ```\n \n- Configure the LUIS applications (HomeAutomation and Weather) required for this sample.\n - Get your [LUIS authoring key](https://docs.microsoft.com/en-us/azure/cognitive-services/LUIS/luis-concept-keys)\n ```bash\n > bf luis:build --in CognitiveModels --authoringKey --botName \n ```\n - Update application settings in `./appsettings.json`. \n \n- Configure the QnA Maker KB required for this sample.\n - Get your [QnA Maker Subscription key](https://docs.microsoft.com/en-us/azure/cognitive-services/QnAMaker/how-to/set-up-qnamaker-service-azure#create-a-new-qna-maker-service)\n ```bash\n > bf qnamaker:build --in CognitiveModels --subscriptionKey --botName \n ```\n - Update kb information in `./appsettings.json`\n \n- Configure Orchestrator to route utterances to LUIS/QnA language services set up above\n - Download Orchestrator intent and entity base models\n ```bash\n > mkdir model\n > bf orchestrator:basemodel:get --out ./model\n > bf orchestrator:basemodel:get --versionId pretrained.20210218.microsoft.dte.00.06.bert_example_ner.en.onnx --out ./model/entity\n ```\n - Create the Orchestrator snapshot\n ```bash\n > mkdir generated\n > bf orchestrator:create --hierarchical --in ./CognitiveModels --out ./generated --model ./model --entityModel ./model/entity\n ```\n The *hierarchical* flag creates top level intents in the snapshot file derived from the .lu/.qna file names in the input folder. As a result, the example utterances are mapped to *HomeAutomation*, *QnAMaker* and *Weather* intents/labels.\n The optional *entityModel* parameter provides path to a pretrained base model for named entity recognition and will be used to compute entity embeddings for Orchestrator snapshot file.\n\n - Verify appsettings.json has the following:\n\n ```\n \"Orchestrator\": {\n \"ModelFolder\": \".\\\\model\",\n \"SnapshotFile\": \".\\\\generated\\\\orchestrator.blu\"\n }\n ```\n \n- Run the bot from a terminal or from Visual Studio, choose option A or B.\n A) From a terminal\n\n ```bash\n > cd samples\\csharp_dotnetcore\\14.nlp-with-orchestrator\n > dotnet run\n ```\n B) Or from Visual Studio\n - Launch Visual Studio\n - File -> Open -> Project/Solution\n - Navigate to `Orchestrator` folder\n - Select `OrchestratorSamples.sln` file\n - Right click on `01.dispatch-bot` project in the solution and 'Set as Startup Project'\n - Press `F5` to run the project\n\n## Connect to the bot using Bot Framework Emulator\n\n- Launch Bot Framework Emulator\n- File -> Open Bot\n- Enter a Bot URL of `http://localhost:3978/api/messages`\n- Try entering several utterances related to home automation, weather or some chit chat. Some examples:\n - set lights to bright\n - good morning\n - how is the weather in seattle\n\n## Further reading\n- [Dispatch Migration Example](https://github.com/microsoft/botframework-sdk/blob/main/Orchestrator/docs/DispatchMigrationExample.md)\n- [Bot Framework Documentation](https://docs.botframework.com)\n- [BF Orchestrator Command Usage](https://github.com/microsoft/botframework-sdk/blob/main/Orchestrator/docs/BFOrchestratorUsage.md)\n- [Bot Basics](https://docs.microsoft.com/azure/bot-service/bot-builder-basics?view=azure-bot-service-4.0)\n- [Activity processing](https://docs.microsoft.com/en-us/azure/bot-service/bot-builder-concept-activity-processing?view=azure-bot-service-4.0)\n- [Azure Bot Service Introduction](https://docs.microsoft.com/azure/bot-service/bot-service-overview-introduction?view=azure-bot-service-4.0)\n- [Azure Bot Service Documentation](https://docs.microsoft.com/azure/bot-service/?view=azure-bot-service-4.0)\n- [.NET Core CLI tools](https://docs.microsoft.com/en-us/dotnet/core/tools/?tabs=netcore2x)\n- [Azure CLI](https://docs.microsoft.com/cli/azure/?view=azure-cli-latest)\n- [Azure Portal](https://portal.azure.com)\n- [Channels and Bot Connector Service](https://docs.microsoft.com/en-us/azure/bot-service/bot-concepts?view=azure-bot-service-4.0)\n\n" }, { "path": "Orchestrator/Samples/dotnet/nlp-with-entities/Startup.cs", "content": "// Copyright (c) Microsoft Corporation. All rights reserved.\n// Licensed under the MIT License.\n\nusing System.IO;\nusing Microsoft.AspNetCore.Builder;\nusing Microsoft.AspNetCore.Hosting;\nusing Microsoft.AspNetCore.Mvc;\nusing Microsoft.Bot.Builder;\nusing Microsoft.Bot.Builder.AI.Orchestrator;\nusing Microsoft.Bot.Builder.Integration.AspNet.Core;\nusing Microsoft.Extensions.Configuration;\nusing Microsoft.Extensions.DependencyInjection;\nusing Microsoft.Extensions.Hosting;\n\nnamespace Microsoft.BotBuilderSamples\n{\n public class Startup\n {\n public Startup(IConfiguration configuration)\n {\n OrchestratorConfig = configuration.GetSection(\"Orchestrator\").Get();\n }\n\n public OrchestratorConfig OrchestratorConfig { get; }\n\n // This method gets called by the runtime. Use this method to add services to the container.\n public void ConfigureServices(IServiceCollection services)\n {\n services.AddMvc().SetCompatibilityVersion(CompatibilityVersion.Latest);\n\n // Create the Bot Framework Adapter with error handling enabled.\n services.AddSingleton();\n\n services.AddSingleton(InitializeOrchestrator());\n\n // Create the bot services (LUIS, QnA) as a singleton.\n services.AddSingleton();\n\n // Create the bot as a transient.\n services.AddTransient();\n }\n\n // This method gets called by the runtime. Use this method to configure the HTTP request pipeline.\n public void Configure(IApplicationBuilder app, IWebHostEnvironment env)\n {\n if (env.IsDevelopment())\n {\n app.UseDeveloperExceptionPage();\n }\n\n app.UseDefaultFiles()\n .UseStaticFiles()\n .UseWebSockets()\n .UseRouting()\n .UseAuthorization()\n .UseEndpoints(endpoints =>\n {\n endpoints.MapControllers();\n });\n }\n\n private OrchestratorRecognizer InitializeOrchestrator()\n {\n string modelFolder = Path.GetFullPath(OrchestratorConfig.ModelFolder);\n string snapshotFile = Path.GetFullPath(OrchestratorConfig.SnapshotFile);\n OrchestratorRecognizer orc = new OrchestratorRecognizer()\n {\n ModelFolder = modelFolder,\n SnapshotFile = snapshotFile\n };\n return orc;\n }\n }\n}\n" }, { "path": "Orchestrator/Samples/dotnet/nlp-with-entities/nlp-with-entities.csproj", "content": "\n\n \n netcoreapp3.1\n latest\n x64\n \n \n \n \n \n \n \n\n \n \n \n \n \n \n\n \n \n PreserveNewest\n \n \n PreserveNewest\n \n \n PreserveNewest\n \n \n PreserveNewest\n \n \n PreserveNewest\n \n \n\n" }, { "path": "Orchestrator/Samples/dotnet/nlp-with-entities/wwwroot/default.html", "content": "\n\n\n\n \n \n Dispatch bot sample with Orchestrator\n \n \n\n\n\n
\n
\n
\n
NLP With Dispatch Sample
\n
\n
\n
\n
\n
Your bot is ready!
\n
You can test your bot in the Bot Framework Emulator
\n by connecting to http://localhost:3978/api/messages.
\n
Download the Emulator
\n
Visit Azure\n Bot Service to register your bot and add it to
\n various channels. The bot's endpoint URL typically looks\n like this:
\n
https://your_bots_hostname/api/messages
\n
\n
\n
\n
\n
\n
\n\n\n" }, { "path": "Orchestrator/docs/BFOrchestratorInteractive.md", "content": "\r\n# Interactive\r\n\r\n[BF Orchestrator CLI][1] has an \"interactive\" command which enables a user to\r\ndynamically interact with an Orchestrator base language model (see examples in [Start an interactive session without a training set](#start-an-interactive-session-without-a-training-set)) and\r\nimprove the accuracy of an existing language model (see examples in [Start an interactive session with a training set](#start-an-interactive-session-with-a-training-set)) through some CLI commandlets.\r\n\r\nDuring an interactive session, the command loads an Orchestrator base model\r\ninto memory and manage several input cache variables that can be used by\r\nensuing commandlets for maintaining the base model's example set. These variables include:\r\n\r\n- **\"current\" utterance** -- A cache for storing an utterance that can be used by several commandlets\r\n for interacting with an Orchestrator model.\r\n- **\"current\" intent labels** -- A cache for storing an array of intent labels that can be used by several\r\n commandlet for interacting with an Orchestrator model.\r\n- **\"new\" intent labels** -- Another cache for storing an array of intent labels, which were mainly\r\n used for changing an utterance's intent labels within an Orchestrator model.\r\n\r\n## Scenarios\r\n\r\n### Start an interactive session without a training set\r\n\r\nAn Orchestrator user can launch the interactive command without a training set.\r\nDuring an session, the user can interactively add utterance examples, revise them, remove them,\r\nvalidate and create an evaluation report, etc..\r\n\r\n#### Argument setup\r\n\r\nBelow is a command snippet for a user to start the interactive command with two arguments:\r\n\r\n - \"--model\" -- folder pointing to an Orchestrator base model\r\n - \"--out\" -- out folder for validation evaluation reports.\r\n\r\n```\r\n> set ORCHESTRATOR_MODEL=\r\n> set EVALUATING_OUTPUT=\r\n> bf orchestrator:interactive --out=%EVALUATING_OUTPUT% --model=%ORCHESTRATOR_MODEL%\r\n```\r\n\r\nAfter executing the command snippet, the command enters an interactive session shown below:\r\n```\r\n> \"Current\" utterance: \"\"\r\n> \"Current\" intent label array: \"\"\r\n> \"New\" intent label array: \"\"\r\nPlease enter a commandlet, \"h\" for help >\r\n```\r\n\r\n#### Cache an utterance\r\n\r\nThe user can then use the 'u' commandlet to enter an utterance and cache it into the \"current\" utterance\r\nvariable. Below is what happened after a user enters \"hi\" as a new utterance using the 'u' commandlet.\r\n\r\n```\r\nPlease enter a commandlet, \"h\" for help > u\r\nPlease enter an utterance > hi\r\n> \"Current\" utterance: \"hi\"\r\n> \"Current\" intent label array: \"\"\r\n> \"New\" intent label array: \"\"\r\nPlease enter a commandlet, \"h\" for help >\r\n```\r\n\r\n#### Cache an intent label\r\n\r\nThen the user can issue the 'i' commandlet to enter a new intent label and cache it into the \"current\"\r\nintent label variable. Notice that Orchestrator supports an utterance with multiple labels, so the \r\nintent variable cache is actually an array for holding multiple intent labels associated with\r\nthe \"current\" utterance. \r\nBelow is what happened after a user enter \"greeting\" as a new intent label using the 'i' commandlet.\r\n\r\n```\r\nPlease enter a commandlet, \"h\" for help > i\r\nPlease enter a \"current\" intent label > greeting\r\n> \"Current\" utterance: \"hi\"\r\n> \"Current\" intent label array: \"greeting\"\r\n> \"New\" intent label array: \"\"\r\nPlease enter a commandlet, \"h\" for help >\r\n```\r\n\r\n#### Add example to Orchestrator core\r\n\r\nAfter a user enter \"hi\" and \"greeting\" during the interactive session, she/he can add this example to\r\nthe Orchestrator core. It will become part of the example set Orchestrator core uses for\r\npredicting the intent for new queries. In a sense, this new example was added to a \"training\" set\r\nfor a supervised machine learning model.\r\nThe commandlet for adding a new example is 'a'.\r\n\r\n```\r\nPlease enter a commandlet, \"h\" for help > a\r\n> Utterance 'hi' has been added to '[\r\n \"greeting\"\r\n]'\r\n> \"Current\" utterance: \"hi\"\r\n> \"Current\" intent label array: \"greeting\"\r\n> \"New\" intent label array: \"\"\r\nPlease enter a commandlet, \"h\" for help >\r\n```\r\n\r\n#### Show statistics of the example set\r\n\r\nOnce a user has entered a sufficient amount of intent/utterance examples into Orchestrator core,\r\nshe/he can then use the 's' commandlet to show some statistics of the example set.\r\n\r\n```\r\nPlease enter a commandlet, \"h\" for help > s\r\n> Per-label #examples: {\r\n \"greeting\": 1\r\n}\r\n> Total #examples:1\r\n> \"Current\" utterance: \"hi\"\r\n> \"Current\" intent label array: \"greeting\"\r\n> \"New\" intent label array: \"\"\r\nPlease enter a commandlet, \"h\" for help >\r\n```\r\n\r\n#### Add examples to current cached intent label\r\n\r\nA user can enter more intent/utterance examples.\r\nBelow is what happened if the user enters and added two more utterances (\"hello\" and \"good morning\")\r\nfor the same \"greeting\" intents. \r\n\r\n```\r\nPlease enter a commandlet, \"h\" for help > u\r\nPlease enter an utterance > hello\r\n> \"Current\" utterance: \"hello\"\r\n> \"Current\" intent label array: \"greeting\"\r\n> \"New\" intent label array: \"\"\r\nPlease enter a commandlet, \"h\" for help > a\r\n> Utterance 'hello' has been added to '[\r\n \"greeting\"\r\n]'\r\n> \"Current\" utterance: \"hello\"\r\n> \"Current\" intent label array: \"greeting\"\r\n> \"New\" intent label array: \"\"\r\nPlease enter a commandlet, \"h\" for help > u\r\nPlease enter an utterance > good morning\r\n> \"Current\" utterance: \"good morning\"\r\n> \"Current\" intent label array: \"greeting\"\r\n> \"New\" intent label array: \"\"\r\nPlease enter a commandlet, \"h\" for help > a\r\n> Utterance 'good morning' has been added to '[\r\n \"greeting\"\r\n]'\r\n> \"Current\" utterance: \"good morning\"\r\n> \"Current\" intent label array: \"greeting\"\r\n> \"New\" intent label array: \"\"\r\nPlease enter a commandlet, \"h\" for help >\r\n```\r\n\r\n#### Add example for a new intent label\r\n\r\nA user can also enter some utterances with different intents.\r\nBelow is what happened if a user enters a new utterance \"good bye\" with \r\na new intent \"farewell.\" However the user forgot to clear the current intent labels, so\r\nthe \"good bye\" intent is actually associated with two intents (\"greeting,farewell\").\r\nEven though this is not completely wrong semantically, but a user can always\r\nclear the intent variable cache (with the 'ci' commandlet). Remember that the utterance and intents have\r\nnot been added to Orchestrator core yet.\r\n\r\n```\r\nPlease enter a commandlet, \"h\" for help > u\r\nPlease enter an utterance > good bye\r\n> \"Current\" utterance: \"good bye\"\r\n> \"Current\" intent label array: \"greeting\"\r\n> \"New\" intent label array: \"\"\r\nPlease enter a commandlet, \"h\" for help > i\r\nPlease enter a \"current\" intent label > farewell\r\n> \"Current\" utterance: \"good bye\"\r\n> \"Current\" intent label array: \"greeting,farewell\"\r\n> \"New\" intent label array: \"\"\r\nPlease enter a commandlet, \"h\" for help >\r\n```\r\n\r\n#### Clear intent label cache\r\n\r\nThe commandlet to clear the intent label cache is 'ci'.\r\nBelow is what happened if the user clear the intent label variable cache.\r\nSimilarly, the utterance variable cache can be cleared by the 'cu' commandlet.\r\n\r\n```\r\nPlease enter a commandlet, \"h\" for help > ci\r\n> \"Current\" utterance: \"good bye\"\r\n> \"Current\" intent label array: \"\"\r\n> \"New\" intent label array: \"\"\r\nPlease enter a commandlet, \"h\" for help >\r\n```\r\n\r\nThe user can then reenter the \"farewell\" intent into the cache and add it to Orchestrator core:\r\n```\r\nPlease enter a commandlet, \"h\" for help > i\r\nPlease enter a \"current\" intent label > farewell\r\n> \"Current\" utterance: \"good bye\"\r\n> \"Current\" intent label array: \"farewell\"\r\n> \"New\" intent label array: \"\"\r\nPlease enter a commandlet, \"h\" for help > a\r\n> Utterance 'good bye' has been added to '[\r\n \"farewell\"\r\n]'\r\n> \"Current\" utterance: \"good bye\"\r\n> \"Current\" intent label array: \"farewell\"\r\n> \"New\" intent label array: \"\"\r\nPlease enter a commandlet, \"h\" for help >\r\n```\r\n\r\nAfter a while, a user can re-issue the 's' commandlet and see the label utterance tabulation:\r\nFrom the session below, we can see that there are three utterances for the \"greeting\" intent,\r\nand one for the \"farewell\" intent.\r\n\r\n```\r\nPlease enter a commandlet, \"h\" for help > s\r\n> Per-label #examples: {\r\n \"greeting\": 3,\r\n \"farewell\": 1\r\n}\r\n> Total #examples:4\r\n> \"Current\" utterance: \"good bye\"\r\n> \"Current\" intent label array: \"farewell\"\r\n> \"New\" intent label array: \"\"\r\nPlease enter a commandlet, \"h\" for help >\r\n```\r\n\r\n#### Make prediction\r\n\r\nAfter there are some intent/utterance entered to Orchestrator core, the user can use\r\nOrchestrator core to make some prediction.\r\nBelow is what happened if \"bye\" is entered as the \"Current\" utterance and use the 'p'\r\ncommandlet for an prediction. Orchestrator core responded with a prediction\r\nof the \"farewell\" intent, a high score of 0.977, and the closest example within the \r\n\"farewell\" intent, \"good bye\".\r\nOn the other hand, the \"greeting\" intent was predicted with a score of 0.691 and\r\nthe closest example was \"good morning.\"\r\n\r\n```\r\nPlease enter an utterance > bye\r\n> \"Current\" utterance: \"bye\"\r\n> \"Current\" intent label array: \"farewell\"\r\n> \"New\" intent label array: \"\"\r\nPlease enter a commandlet, \"h\" for help > p\r\n> Prediction:\r\n[\r\n {\r\n \"closest_text\": \"good bye\",\r\n \"score\": 0.977452505191936,\r\n \"label\": {\r\n \"name\": \"farewell\",\r\n \"label_type\": 1,\r\n \"span\": {\r\n \"length\": 3,\r\n \"offset\": 0\r\n }\r\n }\r\n },\r\n {\r\n \"closest_text\": \"good morning\",\r\n \"score\": 0.6907451700060104,\r\n \"label\": {\r\n \"name\": \"greeting\",\r\n \"label_type\": 1,\r\n \"span\": {\r\n \"length\": 3,\r\n \"offset\": 0\r\n }\r\n }\r\n }\r\n]\r\n> \"Current\" utterance: \"bye\"\r\n> \"Current\" intent label array: \"farewell\"\r\n> \"New\" intent label array: \"\"\r\nPlease enter a commandlet, \"h\" for help >\r\n```\r\n\r\n#### Hints on new intent label needed\r\n\r\nThe same utterance, intent, add, predict commandlets can be repeated for a user\r\nto build a sizeable example set within Orchestrator core.\r\nA user try out some new utterance, such as \"wake me up at 10AM\" and he/she can\r\nsee the prediction based on the two existing intent labels.\r\nAs we can see for the prediction, the prediction scores are fairly low, 0.322 and 0.275.\r\nThe low scores indicate that this new utterance may need a new intent label.\r\n\r\n```\r\nPlease enter an utterance > wake me up at 10AM\r\n> \"Current\" utterance: \"wake me up at 10AM\"\r\n> \"Current\" intent label array: \"farewell\"\r\n> \"New\" intent label array: \"\"\r\nPlease enter a commandlet, \"h\" for help > p\r\n> Prediction:\r\n[\r\n {\r\n \"closest_text\": \"good morning\",\r\n \"score\": 0.322484918357831,\r\n \"label\": {\r\n \"name\": \"greeting\",\r\n \"label_type\": 1,\r\n \"span\": {\r\n \"length\": 18,\r\n \"offset\": 0\r\n }\r\n }\r\n },\r\n {\r\n \"closest_text\": \"good bye\",\r\n \"score\": 0.2750571664975262,\r\n \"label\": {\r\n \"name\": \"farewell\",\r\n \"label_type\": 1,\r\n \"span\": {\r\n \"length\": 18,\r\n \"offset\": 0\r\n }\r\n }\r\n }\r\n]\r\n> \"Current\" utterance: \"wake me up at 10AM\"\r\n> \"Current\" intent label array: \"farewell\"\r\n> \"New\" intent label array: \"\"\r\nPlease enter a commandlet, \"h\" for help >\r\n```\r\n\r\nBelow is what happened if the user enter a new intent label and add the\r\n\"wake me up at 10AM\" utterances to Ochestrator core.\r\n\r\n```\r\nPlease enter a commandlet, \"h\" for help > ci\r\n> \"Current\" utterance: \"wake me up at 10AM\"\r\n> \"Current\" intent label array: \"\"\r\n> \"New\" intent label array: \"\"\r\nPlease enter a commandlet, \"h\" for help > i\r\nPlease enter a \"current\" intent label > alarm\r\n> \"Current\" utterance: \"wake me up at 10AM\"\r\n> \"Current\" intent label array: \"alarm\"\r\n> \"New\" intent label array: \"\"\r\nPlease enter a commandlet, \"h\" for help > a\r\n> Utterance 'wake me up at 10AM' has been added to '[\r\n \"alarm\"\r\n]'\r\n> \"Current\" utterance: \"wake me up at 10AM\"\r\n> \"Current\" intent label array: \"alarm\"\r\n> \"New\" intent label array: \"\"\r\nPlease enter a commandlet, \"h\" for help >\r\n```\r\n\r\nAs we can see from issuing the 's' commandlet, now we have one more intents.\r\n\r\n```\r\nPlease enter a commandlet, \"h\" for help > s\r\n> Per-label #examples: {\r\n \"greeting\": 3,\r\n \"farewell\": 1,\r\n \"alarm\": 1\r\n}\r\n> Total #examples:5\r\n> \"Current\" utterance: \"wake me up at 10AM\"\r\n> \"Current\" intent label array: \"alarm\"\r\n> \"New\" intent label array: \"\"\r\nPlease enter a commandlet, \"h\" for help >\r\n```\r\n\r\n#### Evaluate the example set\r\n\r\nAfter a while, the user can use the 'v' commandlet to evaluate the example set.\r\nBelow is what happened after issuing the 'v' commandlet.\r\nThe user can open the evaluation report file and review the progress thus far.\r\nSince there are very few examples per intent, and the valuation is done in\r\nleave-one-out cross validation (LOOCV), so the report does not look good.\r\n\r\n```\r\nPlease enter a commandlet, \"h\" for help > v\r\n> Leave-one-out cross validation is done and reports generated in 'experiment_evaluating_PrebuildDomain\\orchestrator_predicting_set_summary.html'\r\n> \"Current\" utterance: \"wake me up at 10AM\"\r\n> \"Current\" intent label array: \"alarm\"\r\n> \"New\" intent label array: \"\"\r\nPlease enter a commandlet, \"h\" for help >\r\n```\r\n\r\n#### Remove utterance\r\n\r\nHowever, if the user feels that the \"wake me up at 10AM\" is not what this chatbot is concerned with,\r\nshe/he can remove the utterance using the 'r' commandlet. The 'r' commandlet call Orchestrator core\r\nto remove the intent and utterance example currently cached.\r\nOn the other hand, the user can reference the evaluation report and pick out the utterance\r\nexample and cache it into the \"current\" variables.\r\nBelow is what happened that the user issue the 'vm' (validation/misclassified) commandlet and\r\nenter an index '2' for the No. 2 utterance and put it and its intent to the \"Current\" cache.\r\nSimilarly a user can issue the 'va' (validation/ambiguous), 'vl' (validation/low-confidence), and\r\nthe 'vd' (validation/duplicates) commands to pick up the utterance and intent examples and cache them\r\ninto the variables.\r\n\r\n```\r\nPlease enter a commandlet, \"h\" for help > vm\r\nPlease enter an index from the Misclassified report > 2\r\n> \"Current\" utterance: \"wake me up at 10AM\"\r\n> \"Current\" intent label array: \"alarm\"\r\n> \"New\" intent label array: \"\"\r\nPlease enter a commandlet, \"h\" for help >\r\n```\r\n\r\nBelow is what happened after issuing the 'r' commandlet to remove the \"wake me up at 10AM\"\r\nutterance. Since there is only one utterance for the \"alarm\" intent, the whole intent is\r\neliminated. However, a user can simply issue the 'rl' commandlet to eliminate an intent label\r\nwith all of its utterances.\r\nNow there are two intent left in the Orchestrator core.\r\n```\r\n> \"Current\" utterance: \"wake me up at 10AM\"\r\n> \"Current\" intent label array: \"alarm\"\r\n> \"New\" intent label array: \"\"\r\nPlease enter a commandlet, \"h\" for help > r\r\n> Utterance 'wake me up at 10AM' has been removed from '[\r\n \"alarm\"\r\n]'\r\n> \"Current\" utterance: \"wake me up at 10AM\"\r\n> \"Current\" intent label array: \"alarm\"\r\n> \"New\" intent label array: \"\"\r\nPlease enter a commandlet, \"h\" for help > s\r\n> Per-label #examples: {\r\n \"greeting\": 3,\r\n \"farewell\": 1\r\n}\r\n> Total #examples:4\r\n> \"Current\" utterance: \"wake me up at 10AM\"\r\n> \"Current\" intent label array: \"alarm\"\r\n> \"New\" intent label array: \"\"\r\nPlease enter a commandlet, \"h\" for help >\r\n```\r\n\r\n#### Detailed configuration information\r\n\r\nFor more detailed configuration information for Orchestrator, a user can issue the 'd' commandlet.\r\nBelow is what happened after a user issue the 'd' commandlet, where\r\nthere are several threshold used for the 'v' commandlet during creating an evaluation report.\r\n\r\n```\r\nPlease enter a commandlet, \"h\" for help > d\r\n> Ambiguous closeness: 0.2\r\n> Low-confidence closeness: 0.5\r\n> Multi-label threshold: 1\r\n> Unknown-label threshold: 0.3\r\n> Obfuscation flag: false\r\n> Orchestrator configuration: {\r\n \"CDF\": \"{\\\"zero\\\":0.400000,\\\"unknown\\\":0.700000,\\\"one\\\":1.000000}\",\r\n \"EmbedderVersion\": 1,\r\n \"Framework\": \"onnx\",\r\n \"Layers\": 6,\r\n \"MinRequiredCoreVersion\": \"1.0.0\",\r\n \"ModelFile\": \"D:\\\\testsBfCliOrchestrator\\\\_model\\\\model_bert_6l\\\\model.onnx\",\r\n \"ModelType\": \"dte_bert\",\r\n \"Name\": \"pretrained.20200924.microsoft.dte.00.06.en.onnx\",\r\n \"Publisher\": \"Microsoft\",\r\n \"Version\": \"pretrained.20200924.microsoft.dte.00.06.en.onnx\",\r\n \"VocabFile\": \"D:\\\\testsBfCliOrchestrator\\\\_model\\\\model_bert_6l\\\\vocab.txt\",\r\n \"ignore_same_example\": true,\r\n \"knn_k\": 3,\r\n \"nlr_path\": \"D:\\\\testsBfCliOrchestrator\\\\_model\\\\model_bert_6l\",\r\n \"use_compact_embeddings\": true,\r\n \"use_gap\": false,\r\n \"use_unknown\": true\r\n}\r\n> Current label-index map: {\r\n \"farewell\": 0,\r\n \"greeting\": 1\r\n}\r\n> \"Current\" utterance: \"wake me up at 10AM\"\r\n> \"Current\" intent label array: \"alarm\"\r\n> \"New\" intent label array: \"\"\r\nPlease enter a commandlet, \"h\" for help >\r\n```\r\n\r\n- The \"Ambiguous closeness\" threshold is currently set to 0.2, which means if an utterance\r\n is correctly predicted, but there are some other intent with a score close to 80% of the\r\n ground-truth intent label, then that close utterance will be listed as an \"ambiguous\" prediction\r\n in the evaluation report.\r\n\r\n- The \"Low-confidence\" threshold is default to 0.5, which means that if an utterance's ground-truth\r\n intent label is predicted to have the highest score, but the score is lower than 0.5, then\r\n this utterance will be listed as \"low confidence\"\r\n\r\n- The \"Unknown-label\" threshold is default to 0.3, which means that if the highest predicted\r\n intent score is lower than 0.3, then Orchestrator will simply declare that the intent is unknown.\r\n\r\n- A user can use the 'vo' commandlet to set the \"obfuscate\" flag,\r\n so that the 'v' commandlet can create an evaluation report with all the intent and utterances obfuscated.\r\n An obfuscated report only show some metadata of the example set without any intent or utterance\r\n disclosed, so that the report might be shareable for benchmark and comparison purpose.\r\n The 'd' commandlet also list some internal information of the base model and settings used by \r\n Orchestrator core. These information can be used for debugging purpose.\r\n These thresholds can be reset using the 'vat', 'vlt', 'vmt' and 'vut' commandlets.\r\n\r\n#### Create a new snapshot (.BLU) file\r\n\r\nAfter a user has reached a milestone, she/he can create a new snapshot (.BLU) file using the 'n' command.\r\nIn the future, the user can run an interactive session preloaded with this new snapshot file and\r\nkeep refining the intent and utterance examples.\r\nBelow is what happened after the running the 'n' command and the location of the new snapshot file.\r\n\r\n```\r\nPlease enter a commandlet, \"h\" for help > n\r\n> A new snapshot has been saved to 'experiment_evaluating_PrebuildDomain\\orchestrator_predicting_snapshot_set.blu'\r\n> \"Current\" utterance: \"wake me up at 10AM\"\r\n> \"Current\" intent label array: \"alarm\"\r\n> \"New\" intent label array: \"\"\r\nPlease enter a commandlet, \"h\" for help >\r\n```\r\n\r\n#### Exit the session\r\n\r\nAt the end, a user can enter the 'q' commandlet to exit the session.\r\n\r\n### Start an interactive session with a training set\r\n\r\nSince there is already a .blu snapshot file from the last session, a user can\r\nuse the interactive command to continue refining the example set by using the \"--in\" argument.\r\n\r\n#### Argument setup\r\n\r\n```\r\n> set ORCHESTRATOR_MODEL=\r\n> set EVALUATING_OUTPUT=\r\n> bf orchestrator:interactive --out=%EVALUATING_OUTPUT% --model=%ORCHESTRATOR_MODEL% --in=%EVALUATING_OUTPUT%\\orchestrator_predicting_snapshot_set.blu\r\n```\r\n\r\n#### Show statistics of the example set\r\n\r\nAfter a new interactive session, the user can issue the 's' commandlet and take a look\r\nof the intent/utterance distribution.\r\n\r\n```\r\n> \"Current\" utterance: \"\"\r\n> \"Current\" intent label array: \"\"\r\n> \"New\" intent label array: \"\"\r\nPlease enter a commandlet, \"h\" for help > s\r\n> Per-label #examples: {\r\n \"greeting\": 3,\r\n \"farewell\": 1\r\n}\r\n> Total #examples:4\r\n> \"Current\" utterance: \"\"\r\n> \"Current\" intent label array: \"\"\r\n> \"New\" intent label array: \"\"\r\nPlease enter a commandlet, \"h\" for help >\r\n```\r\n\r\n#### Make predictions on utterances\r\n\r\nThe user can try out some more utterances.\r\n\r\n```\r\nPlease enter a commandlet, \"h\" for help > u\r\nPlease enter an utterance > bye bye\r\n> \"Current\" utterance: \"bye bye\"\r\n> \"Current\" intent label array: \"\"\r\n> \"New\" intent label array: \"\"\r\nPlease enter a commandlet, \"h\" for help > p\r\n> Prediction:\r\n[\r\n {\r\n \"closest_text\": \"good bye\",\r\n \"score\": 0.9852574345531401,\r\n \"label\": {\r\n \"name\": \"farewell\",\r\n \"label_type\": 1,\r\n \"span\": {\r\n \"length\": 7,\r\n \"offset\": 0\r\n }\r\n }\r\n },\r\n {\r\n \"closest_text\": \"good morning\",\r\n \"score\": 0.7110401992969432,\r\n \"label\": {\r\n \"name\": \"greeting\",\r\n \"label_type\": 1,\r\n \"span\": {\r\n \"length\": 7,\r\n \"offset\": 0\r\n }\r\n }\r\n }\r\n]\r\n> \"Current\" utterance: \"bye bye\"\r\n> \"Current\" intent label array: \"\"\r\n> \"New\" intent label array: \"\"\r\nPlease enter a commandlet, \"h\" for help >\r\n```\r\n\r\n#### Add examples to existing language model\r\n\r\nThe predicted intent label was \"farewell\" with a very high score of 0.985. However,\r\nsuppose it's a low score, then the user can improve model\r\nby adding the utterance into the example set.\r\nBelow is what happened after the \"bye bye\" utterance is added.\r\n\r\n```\r\nPlease enter a commandlet, \"h\" for help > i\r\nPlease enter a \"current\" intent label > farewell\r\n> \"Current\" utterance: \"bye bye\"\r\n> \"Current\" intent label array: \"farewell\"\r\n> \"New\" intent label array: \"\"\r\nPlease enter a commandlet, \"h\" for help > a\r\n> Utterance 'bye bye' has been added to '[\r\n \"farewell\"\r\n]'\r\n> \"Current\" utterance: \"bye bye\"\r\n> \"Current\" intent label array: \"farewell\"\r\n> \"New\" intent label array: \"\"\r\nPlease enter a commandlet, \"h\" for help > s\r\n> Per-label #examples: {\r\n \"greeting\": 3,\r\n \"farewell\": 2\r\n}\r\n> Total #examples:5\r\n> \"Current\" utterance: \"bye bye\"\r\n> \"Current\" intent label array: \"farewell\"\r\n> \"New\" intent label array: \"\"\r\nPlease enter a commandlet, \"h\" for help >\r\n```\r\n\r\nBelow is what happened when a user enters a new utterance \"good day\" and\r\nmakes a prediction of it.\r\nAs we can see that the prediction scores for the \"greeting\" and \"farewell\" intents are both very high and close.\r\nThe user then added the \"good day\" utterance to the \"farewell\" intent.\r\n\r\n```\r\nPlease enter a commandlet, \"h\" for help > u\r\nPlease enter an utterance > good day\r\n> \"Current\" utterance: \"good day\"\r\n> \"Current\" intent label array: \"farewell\"\r\n> \"New\" intent label array: \"\"\r\nPlease enter a commandlet, \"h\" for help > p\r\n> Prediction:\r\n[\r\n {\r\n \"closest_text\": \"good morning\",\r\n \"score\": 0.8536692972984349,\r\n \"label\": {\r\n \"name\": \"greeting\",\r\n \"label_type\": 1,\r\n \"span\": {\r\n \"length\": 8,\r\n \"offset\": 0\r\n }\r\n }\r\n },\r\n {\r\n \"closest_text\": \"good bye\",\r\n \"score\": 0.8114553301042285,\r\n \"label\": {\r\n \"name\": \"farewell\",\r\n \"label_type\": 1,\r\n \"span\": {\r\n \"length\": 8,\r\n \"offset\": 0\r\n }\r\n }\r\n }\r\n]\r\n> \"Current\" utterance: \"good day\"\r\n> \"Current\" intent label array: \"farewell\"\r\n> \"New\" intent label array: \"\"\r\nPlease enter a commandlet, \"h\" for help > a\r\n> Utterance 'good day' has been added to '[\r\n \"farewell\"\r\n]'\r\n> \"Current\" utterance: \"good day\"\r\n> \"Current\" intent label array: \"farewell\"\r\n> \"New\" intent label array: \"\"\r\nPlease enter a commandlet, \"h\" for help > s\r\n> Per-label #examples: {\r\n \"greeting\": 3,\r\n \"farewell\": 3\r\n}\r\n> Total #examples:6\r\n> \"Current\" utterance: \"good day\"\r\n> \"Current\" intent label array: \"farewell\"\r\n> \"New\" intent label array: \"\"\r\nPlease enter a commandlet, \"h\" for help >\r\n```\r\n\r\n#### Relabel utterances\r\n\r\nHowever, the user may later decide to change the intent for \"good day\" from \"farewell\" to \"greeting\".\r\nShe/he can them use the 'ni' commandlet to enter \"greeting\" into the \"New\" intent label\r\ncache. After that, the user can issue the 'c' commandlet to change the intent label for \"good day\".\r\nAs we can see below, the label/utterance distribution has changed afterward.\r\nNotice that the user can use the 'cni' commandlet to clear the \"New\" intent label cache.\r\n\r\n```\r\nPlease enter a commandlet, \"h\" for help > s\r\n> Per-label #examples: {\r\n \"greeting\": 3,\r\n \"farewell\": 3\r\n}\r\n> Total #examples:6\r\n> \"Current\" utterance: \"good day\"\r\n> \"Current\" intent label array: \"farewell\"\r\n> \"New\" intent label array: \"\"\r\nPlease enter a commandlet, \"h\" for help > ni\r\nPlease enter a \"new\" intent label > greeting\r\n> \"Current\" utterance: \"good day\"\r\n> \"Current\" intent label array: \"farewell\"\r\n> \"New\" intent label array: \"greeting\"\r\nPlease enter a commandlet, \"h\" for help > c\r\n> Utterance 'good day' has been moved from '[\r\n \"farewell\"\r\n]' to '[\r\n \"greeting\"\r\n]'\r\n> \"Current\" utterance: \"good day\"\r\n> \"Current\" intent label array: \"farewell\"\r\n> \"New\" intent label array: \"greeting\"\r\nPlease enter a commandlet, \"h\" for help > s\r\n> Per-label #examples: {\r\n \"greeting\": 4,\r\n \"farewell\": 2\r\n}\r\n> Total #examples:6\r\n> \"Current\" utterance: \"good day\"\r\n> \"Current\" intent label array: \"farewell\"\r\n> \"New\" intent label array: \"greeting\"\r\nPlease enter a commandlet, \"h\" for help >\r\n```\r\n\r\n#### Check existence of an utterance in the language model\r\n\r\nAlso, the user can enter an utterance and see if it's in the Orchestrator Core's example set or not\r\nusing the 'f' commandlet.\r\nBelow is what happened when a user tries to find out if \"good evening\" is in the example set of not.\r\n\r\n```\r\nPlease enter a commandlet, \"h\" for help > u\r\nPlease enter an utterance > good evening\r\n> \"Current\" utterance: \"good evening\"\r\n> \"Current\" intent label array: \"farewell\"\r\n> \"New\" intent label array: \"greeting\"\r\nPlease enter a commandlet, \"h\" for help > f\r\n> The \"current\" utterance 'good evening' is not in the model.\r\n> \"Current\" utterance: \"good evening\"\r\n> \"Current\" intent label array: \"farewell\"\r\n> \"New\" intent label array: \"greeting\"\r\nPlease enter a commandlet, \"h\" for help >\r\n```\r\n\r\n## List of interactive commandlets\r\nAt any time, if a user needs to reference help messages for the many commandlets, he/she can issue\r\nthe 'h' command for a brief help page on all the commandlets.\r\nBelow is the list of the commandlets that can be issued during a 'interactive' session.\r\n\r\n### 'h' - help command\r\n h - print the help messages with short description of each command\r\n\r\n### 'q' - quit command\r\n q - quit\r\n\r\n### 'd' - display command\r\n d - display the utterance, intent label array input caches, Orchestrator config,\r\n and the label-index map\r\n\r\n### 's' - statistics command\r\n s - show label-utterance statistics of the model examples\r\n\r\n### 'u' - utterance command\r\n u - enter a new utterance and save it as the \"current\" utterance input\r\n\r\n### 'cu' - clear utterance variable command\r\n cu - clear the \"current\" utterance input cache\r\n\r\n### 'i' - intent variable command\r\n i - enter an intent and add it to the \"current\" intent label array input\r\n (can be an index for retrieving a label from the label-index map)\r\n\r\n### 'ci' - clear intent variable command\r\n ci - clear the \"current\" intent label array input cache\r\n\r\n### 'ni' - new intent command\r\n ni - enter an intent and add it to the \"new\" intent label array input\r\n (can be an index for retrieving a label from the label-index map)\r\n\r\n### 'cni' - clear new intent command\r\n cni - clear the \"new\" intent label array input cache\r\n\r\n### 'f' - find command\r\n f - find the \"current\" utterance if it is already in the model example set\r\n\r\n### 'p' - predict command\r\n p - make a prediction on the \"current\" utterance input\r\n\r\n### 'v' - validate command\r\n v - validate the model and save analyses (validation report) to\r\n \"experiment_predicting_va\\orchestrator_predicting_set_summary.html\"\r\n\r\n### 'vd' - utterance/labels retrieval command (from the Duplicates section of the validation report)\r\n vd - reference a validation Duplicates report\r\n (previously generated by the \"v\" command) and enter an index\r\n for retrieving utterance/intents into \"current\"\r\n\r\n### 'va' - utterance/labels retrieval command (from the Ambiguous section of the validation report)\r\n va - reference a validation Ambiguous report\r\n (previously generated by the \"v\" command) and enter an index\r\n for retrieving utterance/intents into \"current\"\r\n\r\n### 'vm' - utterance/labels retrieval command (from the Misclassified section of the validation report)\r\n vm - reference a validation Misclassified report and enter an index\r\n (previously generated by the \"v\" command)\r\n for retrieving utterance/intents into \"current\"\r\n\r\n### 'vl' - utterance/labels retrieval command (from the Low Confidence section of the validation report)\r\n vl - reference a validation LowConfidence report\r\n (previously generated by the \"v\" command) and enter an index\r\n for retrieving utterance/intents into \"current\"\r\n\r\n### 'vat' - ambiguous threshold command\r\n vat - enter a new validation-report ambiguous closeness threshold\r\n\r\n### 'vlt' - low-confidence threshold command\r\n vlt - enter a new validation-report low-confidence threshold\r\n\r\n### 'vmt' - multi-label threshold command\r\n vmt - enter a new multi-label threshold\r\n\r\n### 'vut' - unknown label threshold command\r\n vut - enter a new unknown-label threshold\r\n\r\n### 'vo' - obfuscate command\r\n vo - enter a boolean for obfuscating labels/utterances or not in evaluation reports\r\n generated by the \"v\" command'\r\n\r\n### 'a' - add command\r\n a - add the \"current\" utterance and intent labels to the model example set\r\n\r\n### 'r' - remove command\r\n r - remove the \"current\" utterance and intent labels from the model example set\r\n\r\n### 'c' - change command\r\n c - remove the \"current\" utterance's intent labels and then\r\n add it with the \"new\" intent labels to the model example set\r\n\r\n### 'rl' - remove label command\r\n rl - remove the \"current\" intent labels from the model example set\r\n\r\n### 'n' - new snapshot command\r\n n - create a new snapshot of model examples and save it to\r\n \"experiment_predicting_va\\orchestrator_predicting_training_set.blu\"\r\n\r\n## References\r\n\r\n- [BF Orchestrator CLI](https://aka.ms/bforchestratorcli)\r\n\r\n[1]:https://aka.ms/bforchestratorcli\t\"BF Orchestrator CLI\"\r\n" }, { "path": "Orchestrator/docs/BFOrchestratorReport.md", "content": "# Report Interpretation\r\n\r\nThe [BF Orchestrator CLI][1] has a \"test\" command for evaluating the performance of an Orchestrator snapshot file (with .blu extension). A snapshot is composed of natural language representation base model (see [models][3]) along with a set of examples provided in a label file (typically a [.lu file][4]). The snapshot file is used in Bot Framework to detect intents from user utterances. \r\n\r\nIn order to achieve high quality natural language processing (e.g. intent detection), it is necessary to assess and refine the quality of the model. This is much simplified in Orchestrator because of its use of pre-trained models. The optimization cycle is required in order to account for human language variations. \r\nLearn more about Machine Learning evaluation methodology in the [References](# references) section below.\r\n\r\nTo interpret the report, use the following guidance to actions to improve the snapshot file:\r\n- Incase utterances are the same or semantically similar, mrerge the two intent labels.\r\n- If the utterances are not semantically similar, split an intent's utterance pool and create a new intent label.\r\n- It the utterance are semantically close to a different intent label, change the intent label.\r\n- Rephrase an utterance and make it semantically closer to other utterances labled with the same intent.\r\n- Add more utterances to an intent label as the intent's utterance pool could be too scarce.\r\n- Remove some utterances from an intent label if there were too many utterances label to it.\r\n\r\n# Report Organization\r\n\r\nThe test command thus produces a folder with HTML report and a few supporting artifacts as follows:\r\n\r\n- orchestrator_testing_set_ground_truth_instances.json: test instance ground-truth file in JSON format.\r\n- orchestrator_testing_set_labels.txt: intent labels in a plain text file.\r\n- orchestrator_testing_set_prediction_instances.json: test instance prediction file in JSON format.\r\n- orchestrator_testing_set_scores.txt: test instance prediction file in a plain TSV format.\r\n- orchestrator_testing_set_summary.html: report summary in HTML format\r\n\r\nThe report summary contains several sections as follows:\r\n\r\n## Intent / Utterance Statistics\r\n\r\nThis section contains label and utterance distributions.\r\n\r\nIt has two statistical sections, one for labels, the other utterances. Attached is an example rendition of the section.\r\n\r\n- Label statistics\r\n- Utterance statistics\r\n\r\n![Evaluation Report Intent/Utterance Statistics](media/EvaluationReportTabVaIntentUtteranceStatistics.png)\r\n\r\n### Label statistics\r\n\r\nLabel statistics lists the number of utterances labeled to each label. Additional metrics include utterance prevalence (ratio) for every label. The distributions can give Orchestrator users an overall view of the labels and utterances, and whether the distributions are skewed and emphasize too much on some labels, but not others. A machine learn model may learn more from a label (intent) with more instances (utterances) labeled to it. Thus, a developer can check this table and see if some intents needs more utterances in the snapshot file.\r\n\r\n### Utterance statistics\r\n\r\nOn the other hand, utterance statistics focus on the #label distribution by the utterances. Some utterances are labeled with more than one intents, which might not be desirable and could be a bug. This table reflects the distribution of multi-label utterances.\r\nFrom the above screen snapshot, we can see that there are two utterances labeled twice with distinct labels. Those multi-label utterances will be listed in the next section and the owner can decide to remove the duplicates from the snapshot file.\r\n\r\n## Utterance Duplicates\r\n\r\nSometimes some dataset might contain utterances tagged with the same labels multiple times.\r\nThe report also lists this redundancy.\r\nThis section has two sub-sections:\r\n\r\n- Multi-label utterances and their labels\r\n- Duplicate utterance and label pairs\r\n\r\nThey report on utterances with duplicate or multiple labels. A duplicate utterance is detected when it is present more than once in a snapshot file. Sometimes some dataset might even contain utterances tagged with the same labels multiple times.\r\n\r\nThe report also lists the redundancy of label/utterance pairs. Orchestrator will deduplicate such redundancy.\r\nPlease see the attached screen snapshot as an example.\r\n\r\n![Evaluation Report Utterance Duplicates](media/EvaluationReportTabVaUtteranceDuplicates.png)\r\n\r\n## Ambiguous\r\n\r\nThis section reports on utterances with ambiguous predictions. For an evaluation utterance, if an Orchestrator model correctly predicts its intent label, then it's a true positive instance. Every intent label will be predicted with a score, which is essentially the probability or confidence for that label prediction. The predicted intent usually is the one with the highest score. If the Orchestrator model also makes some other high-score prediction close to that of the correctly predicted label, then we call such a prediction \"ambiguous.\"\r\n\r\nIn this section, the report lists all the utterances with an ambiguous prediction in a table.\r\nThe table has several columns:\r\n\r\n- Utterance -- the utterance\r\n- Labels -- the true labels for the utterance\r\n- Predictions -- the labels predicted by the Orchestrator model\r\n- Close Predictions -- some other labels predicted with a close, high score to that of the predicted label. \r\n\r\nBesides the prediction score, the report also shows the closest example to the utterance\r\nwithin the label's utterance set.\r\nBelow is a screen snapshot of an ambiguous report:\r\n\r\n![Evaluation Report Ambiguous](media/EvaluationReportTabVaAmbiguous.png)\r\n\r\nAmbiguous utterances can be a sign of overlapping intent labels. In another word, two intent labels may have their utterance pools semantically close to each other. In the example report about, the utterance \"what my events today\" was correctly predicted with the Calendar intent label and the closest example is\r\n\"What is on my calendar today\". However the FAQ3 intent was also predicted with a high school and the closest example from that intent is \"What's going on today.\" From these two closest example, one can remove the latter from the FAQ3 intent as it is now more specific to the Calendar intent, then just a FAQ example.\r\n\r\n## Misclassified\r\n\r\nThis section reports on utterances with incorrect predictions. A misclassified predication is one where an Orchestrator model falsely predicts its intent label. Usually the label with the highest prediction score is chosen as the predicted label, but it can be different from the ground-truth label for the utterance.\r\n\r\nSimilar to the last section, the report also lists the prediction and ground-truth labels with\r\ntheir prediction scores and closest examples. Below is a screen snapshot of the misclassified report. A user can also follow the utterances list and decide to update their intent labels and or rephrase the utterances themselves.\r\n\r\n![Evaluation Report Misclassified](media/EvaluationReportTabVaMisclassified.png)\r\n\r\n## Low Confidence\r\n\r\nThis section reports on predictions that scored too low to be considered \"confident\" intent detection.\r\n\r\nSometimes a prediction may be predicted correctly with the highest scores among all labels, but the score is very low, lower than a threshold. We call such predictions low confidence.\r\nNotice that there are several default thresholds used to guide producing the report sections thus far. Usually an Orchestrator user can just predict a label with the highest score, but he/she can also utilize some thresholds for a fine-tuned prediction in a chat bot's dialog logic.\r\n\r\nJust like the last sections, the report lists the prediction and ground-truth labels with their prediction scores and closest examples. Also like the previous sections, the utterances listed in this section can guide a user how to improve the snapshot file.\r\n\r\n![Evaluation Report Low Confidence](media/EvaluationReportTabVaLowConfidence.png)\r\n\r\n## Metrics\r\n\r\nThe Metrics section is an advanced report that contains analytics that is common in Machine Learning evaluation methodologies.\r\n\r\nAdvanced machine-learning practitioners may analyze the overall model performance expressed in machine learning metrics. In this section, the report calculates some common metrics in two sections:\r\n\r\n- Confusion matrix metrics\r\n- Average confusion matrix metrics\r\n\r\n![Evaluation Report Metrics](media/EvaluationReportTabVaMetrics.png)\r\n\r\n### Confusion matrix metrics\r\n\r\nIn this table, the Orchestrator CLI test command reads an evaluation set with ground-truth labels. An evaluation set contains a collection of utterances and their labels. It then calls the Orchestrator base model and makes a prediction for every utterance in the set and generate predicted labels for every utterance. It then compares the predicted labels against the ground-truth labels and creates a table of per-label binary confusion matrices.\r\n\r\nFor a binary confusion matrix, there are four cells:\r\n - #True Positives (TP)\r\n - #False Positives (FP)\r\n - #False Neagatives (FN)\r\n - #True Negatives (TN)\r\n\r\nFor a label, if it exists in an utterance's ground-truth label set as well as in the predicted label set,\r\nthen that utterance is a TP for the label.\r\nIf the label only exists in the utterance's predicted label set, then it's a false positive.\r\nIf the label only exists in the utterance's ground-truth set, then it's a false negative.\r\nIf the label does not exist in either the ground-truth or predicted set, then it's a true negative.\r\n\r\nBy the way, for entity extraction, as a label contains an entity name, entity offset in an utterance and its length,\r\nthere can be numerous true negatives, since there are limitless combinations of the entity attributes\r\nnot in the ground-truth or the predicted sets.\r\nThus, for intent prediction evaluation, all four cells are used to calculate confusion matrix metrics,\r\nbut only the first three are used for entity-extraction evaluation.\r\n\r\nUsing just the first three cells of a binary confusion matrix,\r\nthe Orchestrator \"test\" command can then calculate some\r\nmore sophisticated metrics, including\r\n\r\n - Precision - TP / (TP + FP)\r\n - Recall - TP / (TP + FN)\r\n - F1 - harmonic mean of precision and recall\r\n\r\nThese three metrics do not use TN, but the simple accuracy metric need all 4 cells, including TN.\r\n\r\nFor details of many confusion matrix metrics, please reference [Wikipedia: Confusion matrix][2].\r\n\r\n### Average confusion matrix metrics\r\n\r\nSince Orchestrator can evaluate multiple labels, one confusion matrix for each, there can be many\r\nmetrics for detailed analysis. For reporting and KPI purpose, it would be great to aggregate all\r\nthese metrics for an overall metric and model performance.\r\n\r\nThere are many nuanced ways to aggregate confusion matrix metrics. For comparing models, it's critical\r\nto compare based on a consistent formula. Please reference the [BF Orchestrator CLI][1] readme page for advanced CLI usage details.\r\n\r\n## Thresholds\r\n\r\nThis evaluation report is created using several thresholds that they can also be useful in building chat bot logic. These thresholds can be reset through environment variables listed below:\r\n\r\n- ambiguousClosenessThreshold: default to 0.2, which means that if there are labels predicted to have a score close to within 20% of the top and corrected predicted label, then this utterance is ambiguously predicted.\r\n- lowConfidenceScoreThreshold: default to 0.5, which means that if the top predicted score is lower than 0.5, then the prediction is considered low confidence.\r\n- multiLabelPredictionThreshold: default to 1, which means that the report will predict only one label. However, if the threshold is lower than 1, then every label with a predicted score higher than that will be predicted. This threshold is usually used for multi-label, multi-intent scenarios.\r\n- unknownLabelPredictionThreshold:default to 0.3, which means that the evaluation process will consider a prediction UNKNOWN if the score is lower than that threshold.\r\n\r\n## References\r\n\r\n- [BF Orchestrator CLI](https://aka.ms/bforchestratorcli)\r\n- [Wikipedia: Confusion matrix](https://en.wikipedia.org/wiki/Confusion_matrix)\r\n- [Wikipedia: Training, validation, and test sets](https://en.wikipedia.org/wiki/Training,_validation,_and_test_sets)\r\n- [Machine Learning Mastery](https://machinelearningmastery.com/difference-test-validation-datasets/).\r\n\r\n[1]:https://aka.ms/bforchestratorcli\t\"BF Orchestrator CLI\"\r\n[2]:https://en.wikipedia.org/wiki/Confusion_matrix\t\"Wikipedia: Confusion matrix\"\r\n[3]:https://aka.ms/nlrmodels\t\"NLR Models\"\r\n[4]:https://docs.microsoft.com/en-us/azure/bot-service/file-format/bot-builder-lu-file-format?view=azure-bot-service-4.0 \".LU format\"\r\n[5]: https://github.com/microsoft/botframework-cli/tree/main/packages/orchestrator#bf-orchestratortest \"bf orchestrator:test\"\r\n" }, { "path": "Orchestrator/docs/BFOrchestratorUsage.md", "content": "# BF Orchestrator Command Usage\n\n[BF Orchestrator Command ][1] is used to manage Orchestrator language understanding assets - retrieve, create, evaluate, and tune Orchestrator models to be used in bot development. Typical use is for intent routing to the appropriate language service, LUIS or QnA Maker for additional language tasks such as entity extraction or answer retrieval. This use was previously enabled by use of the legacy *dispatch* CLI. In addition, intent routing can be used to delegate conversations to a backend bot skill. Entity recognition is on the roadmap as a future capability.\n\n## Primary Scenario\n\nThe mainstream bot language recognition development cycle with Orchestrator is generally as follows:\n\n1. Create Intent label / utterance examples file in [.lu format][2] (will be referred to as the *label file*). \n2. Download base model.\n3. Combine the label file .lu from (1) with the base model from (2) to create a .blu file (will be referred to as the *snapshot* file). \n4. Test and refine quality of utterance to intent recognition.\n5. Integrate Orchestrator language recognizer in your bot.\n\n## Steps\n\nWe will use the primary workflow to illustrate how to use Orchestrator commands for full development cycle.\n\n### 1. Create Intent label / utterance examples file in [.lu format][2]\n\nIf you are developing a new language model, simply refer to [Language understanding][8] documentation to author label files. Depending on how you plan to design your bot, you may use a single label file corresponding to a single snapshot file for the entire bot or multiple pairs, each for every adaptive dialog where you plan to use Orchestrator as a recognizer. \n\nIn case of **migration from legacy dispatch**, simply add LUIS app(s) or QnAMaker kb(s) as Orchestrator data sources. Refer to the Dispatch migration documentation [here][11] for more information.\n\nExample:\n```\n> bf orchestrator:add -t luis --id --endpoint -k -v --routingName l_\n> bf orchestrator:add -t qna --id -k --routingName q_\n```\n\n### 2. Download base model\n\nCreate a new folder, say *models*, and download the default base model using: \n\n```\nbf orchestrator:basemodel:get --out ./models\n```\n\nout parameter is optional. If not specified, base model files will be downloaded to the current working directory.\n\nSee also the ```orchestrator:basemodel:list``` command if you wish to download and experiment with different base models. (see descriptions [here][4] ).\n\n### 3. Combine the label file .lu from (1) with the base model from (2) to create a .blu file\n\nThere are two ways to create Orchestrator snapshot file(s), depending on the usage scenarios:\n\n- **To route utterances to LUIS/QnA language services**\n\n Use ```bf orchestrator:create``` command to combine the base model with the label file(s) to create snapshot file for use by the orchestrator recognizer. If using a single folder, create it prior, say *generated*, and specify it in --out parameter:\n\n ```\n bf orchestrator:create --model --in