Repository: microsoft/botframework-sdk Branch: main Commit: ed8adc25069b Files: 790 Total size: 4.8 MB Directory structure: gitextract_dp4i3952/ ├── .editorconfig ├── .gitattributes ├── .github/ │ ├── CODEOWNERS │ ├── ISSUE_TEMPLATE/ │ │ ├── bot-framework-bug.md │ │ └── bot-framework-feature-request.md │ ├── PULL_REQUEST_TEMPLATE.md │ └── issue_template.md ├── .gitignore ├── CODE_OF_CONDUCT.md ├── Common/ │ ├── Telemetry/ │ │ └── README.md │ ├── Transcripts/ │ │ ├── CoreExtensionsTests/ │ │ │ ├── ConversationStateTest.chat │ │ │ ├── CustomStateTest.chat │ │ │ └── UserStateTest.chat │ │ ├── CoreTests/ │ │ │ └── BotAdapted_Bracketing.chat │ │ ├── DialogsTests/ │ │ │ ├── AttachmentPrompt.chat │ │ │ ├── ChoicePrompt.chat │ │ │ ├── ConfirmPrompt.chat │ │ │ ├── ContentAttachment.json │ │ │ ├── DateTimePrompt.chat │ │ │ ├── NumberPrompt.chat │ │ │ ├── TextPrompt.chat │ │ │ ├── Waterfall.chat │ │ │ ├── WaterfallNested.chat │ │ │ └── WaterfallPrompt.chat │ │ ├── LuisTests/ │ │ │ └── LuisMiddleware.chat │ │ ├── QnATests/ │ │ │ └── QnAMiddleware.chat │ │ ├── README.md │ │ └── TranslationTests/ │ │ ├── LocaleConvertToEnglish.chat │ │ ├── TranslateToEnglish.chat │ │ └── TranslateToUserLanguage.chat │ └── badges/ │ └── BotBuilderBadge.md ├── Contributing.md ├── LICENSE ├── Orchestrator/ │ ├── README.md │ ├── Samples/ │ │ ├── CLI/ │ │ │ └── ModelTuning/ │ │ │ ├── README.md │ │ │ ├── common.fixed.lu │ │ │ ├── common.lu │ │ │ ├── common.test.lu │ │ │ └── demo.cmd │ │ └── dotnet/ │ │ └── nlp-with-entities/ │ │ ├── .gitignore │ │ ├── AdapterWithErrorHandler.cs │ │ ├── BotServices.cs │ │ ├── Bots/ │ │ │ └── DispatchBot.cs │ │ ├── CognitiveModels/ │ │ │ ├── HomeAutomation.lu │ │ │ ├── QnAMaker.qna │ │ │ └── Weather.lu │ │ ├── Controllers/ │ │ │ └── BotController.cs │ │ ├── IBotServices.cs │ │ ├── OrchestratorConfig.cs │ │ ├── Program.cs │ │ ├── README.md │ │ ├── Startup.cs │ │ ├── nlp-with-entities.csproj │ │ └── wwwroot/ │ │ └── default.html │ ├── docs/ │ │ ├── BFOrchestratorInteractive.md │ │ ├── BFOrchestratorReport.md │ │ ├── BFOrchestratorUsage.md │ │ ├── DispatchMigrationExample.md │ │ ├── FAQ.md │ │ ├── LICENSE.md │ │ ├── LUFormat.md │ │ ├── NLRModels.md │ │ ├── Orchestrator Intro.pptx │ │ ├── Overview.md │ │ └── package/ │ │ └── README.md │ ├── v0.1/ │ │ ├── nlr_versions.json │ │ └── nlr_versions.schema │ └── v0.2/ │ ├── nlr_versions.json │ └── nlr_versions.schema ├── README.md ├── SECURITY.md ├── codeql3000.yml ├── docs/ │ ├── BF Labeling Guide.md │ └── umlDiagrams/ │ ├── ActivityFlow/ │ │ ├── BFServiceAndBotDetails/ │ │ │ └── BFServiceAndBotDetails_DiagramSource.md │ │ ├── BotFrameworkAdapterFlow/ │ │ │ └── BFAdapterActivityFlow_DiagramSource.md │ │ ├── CustomAdapterFlow/ │ │ │ ├── BufferingWithCustomAdapter_DiagramSource.md │ │ │ └── CustomAdapterActivityFlow_DiagramSource.md │ │ ├── DetailedSdkFlow/ │ │ │ ├── DetailedSdkFlow.md │ │ │ └── DetailedSdkFlow_DiagramSource.md │ │ ├── GeneralActivityFlow.md │ │ ├── GeneralActivityFlow_DiagramSource.md │ │ └── README.md │ └── Authentication/ │ ├── AddingAuthenticationToYourBot/ │ │ ├── AddAuthenticationToYourBotViaABS/ │ │ │ ├── DetailedView_DiagramSource.md │ │ │ └── HigherLevel_DiagramSource.md │ │ └── UserAuthenticationWithinAConversation/ │ │ ├── AuthenticationDocExample_Goal_DiagramSource.md │ │ ├── AuthenticationDocExample_OAuthFlow_HasTokenInStorage_DiagramSource.md │ │ └── AuthenticationDocExample_OAuthFlow_NoTokenToStart_DiagramSource.md │ ├── AuthArchitectureInBotFramework/ │ │ └── OAuthPrompt/ │ │ ├── AdapterSavedAsTurnContextMember_DiagramSource.md │ │ ├── BotFrameworkAdapterCreatesOAuthClient_DiagramSource.md │ │ ├── BotFrameworkAdapterCreatesTokenApiClient_DiagramSource.md │ │ ├── BuildAppCredentialsParticipants_DiagramSource.md │ │ ├── CSharp_AppCredentialsClassDiagram_DiagramSource.md │ │ ├── CSharp_OAuthPromptAndTokenProvider_DiagramSource.md │ │ ├── CSharp_UseAppCredentialsToCreateOAuthClient_DiagramSource.md │ │ ├── CreatingOAuthClientInMoreDetail.md │ │ ├── CreatingOAuthClientInMoreDetail_DiagramSource.md │ │ ├── GetUserTokenSequenceDiagram.md │ │ ├── JS_AppCredentialsClassDiagram_DiagramSource.md │ │ ├── JS_OAuthPromptAndTokenProvider_DiagramSource.md │ │ ├── JS_UseAppCredentialsToCreateTokenApiClient_DiagramSource.md │ │ ├── OAuthClientClassDiagram_DiagramSource.md │ │ ├── OAuthPromptClassDiagrams.md │ │ ├── ProcessActivityCreatesTurnContext_DiagramSource.md │ │ ├── TakesAdapterFromTurnContext_DiagramSource.md │ │ └── TokenApiClientClassDiagram_DiagramSource.md │ ├── ChannelToBot/ │ │ ├── AuthenticateRequestOverview_DiagramSource.md │ │ ├── AuthenticateRequest_DiagramSource.md │ │ ├── ChannelActingOnBehalfOfSelf_DiagramSource.md │ │ ├── ChannelActivingOnBehalfOfUser_DiagramSource.md │ │ ├── ChannelToABSLayerToBot_DiagramSource.md │ │ ├── HigherLevel_TokenAuthentication_DiagramSource.md │ │ ├── JwtTokenValidationClassDiagram_DiagramSource.md │ │ ├── README.md │ │ ├── SimpleChannelWithTokenToBot_DiagramSource.md │ │ ├── TokenSignatureParticipants_DiagramSource.md │ │ └── WhyHelloBot_DiagramSource.md │ ├── HighLevelAuthFlow/ │ │ ├── AuthServerAuthenticatesUser_DiagramSource.md │ │ ├── AuthServerIssuesTokenToBot_DiagramSource.md │ │ ├── GiveBotAccessToProtectedResource_DiagramSource.md │ │ └── HighLevelAuthFlowSequenceDiagram_DiagramSource.md │ ├── OAuthPrompt/ │ │ ├── OAuthPrompt_BeginDialog_DiagramSource.md │ │ ├── OAuthPrompt_ContinueDialog_DetailedView_DiagramSource.md │ │ ├── OAuthPrompt_ContinueDialog_HigherLevel_DiagramSource.md │ │ └── README.md │ └── README.md ├── dri/ │ ├── .gitignore │ ├── README.md │ ├── helpers.py │ ├── output.py │ ├── report.py │ └── requirements.txt ├── parsers/ │ └── LU/ │ ├── ANTLR/ │ │ ├── LUFileLexer.g4 │ │ └── LUFileParser.g4 │ └── JS/ │ ├── build/ │ │ ├── bf-cli-build-test-steps.yml │ │ ├── botframework-cli-azure-devops.yml │ │ ├── botframework-cli-beta.yml │ │ ├── botframework-cli-daily.yml │ │ ├── botframework-cli-mac.yml │ │ ├── botframework-cli-rc.yml │ │ ├── botframework-cli-version.yml │ │ └── botframework-cli.yml │ ├── common/ │ │ ├── config/ │ │ │ └── rush/ │ │ │ ├── .npmrc │ │ │ ├── command-line.json │ │ │ ├── common-versions.json │ │ │ ├── experiments.json │ │ │ ├── pnpmfile.js │ │ │ └── version-policies.json │ │ └── scripts/ │ │ ├── install-run-rush.js │ │ ├── install-run-rushx.js │ │ ├── install-run.js │ │ └── version-and-pack.js │ ├── packages/ │ │ └── lu/ │ │ ├── .editorconfig │ │ ├── .gitignore │ │ ├── .nycrc │ │ ├── README.md │ │ ├── package.json │ │ ├── src/ │ │ │ ├── index.ts │ │ │ ├── parser/ │ │ │ │ ├── composerindex.js │ │ │ │ ├── converters/ │ │ │ │ │ ├── helpers/ │ │ │ │ │ │ └── writer.js │ │ │ │ │ ├── luistocsconverter.js │ │ │ │ │ └── luistotsconverter.js │ │ │ │ ├── cross-train/ │ │ │ │ │ ├── confighelper.js │ │ │ │ │ ├── cross-train.js │ │ │ │ │ └── crossTrainer.js │ │ │ │ ├── index.js │ │ │ │ ├── lu/ │ │ │ │ │ ├── lu.js │ │ │ │ │ ├── luMerger.js │ │ │ │ │ ├── luOptions.js │ │ │ │ │ ├── qna.js │ │ │ │ │ └── qnaOptions.js │ │ │ │ ├── lubuild/ │ │ │ │ │ ├── builder.ts │ │ │ │ │ ├── core.ts │ │ │ │ │ ├── cross-trained-recognizer.ts │ │ │ │ │ ├── multi-language-recognizer.ts │ │ │ │ │ ├── recognizer.ts │ │ │ │ │ └── settings.ts │ │ │ │ ├── lufile/ │ │ │ │ │ ├── LUFileLexer.g4 │ │ │ │ │ ├── LUFileParser.g4 │ │ │ │ │ ├── baseSection.js │ │ │ │ │ ├── classes/ │ │ │ │ │ │ ├── filesToParse.js │ │ │ │ │ │ ├── hclasses.js │ │ │ │ │ │ └── parserObject.js │ │ │ │ │ ├── diagnostic.js │ │ │ │ │ ├── entitySection.js │ │ │ │ │ ├── generated/ │ │ │ │ │ │ ├── LUFileLexer.interp │ │ │ │ │ │ ├── LUFileLexer.js │ │ │ │ │ │ ├── LUFileLexer.tokens │ │ │ │ │ │ ├── LUFileParser.interp │ │ │ │ │ │ ├── LUFileParser.js │ │ │ │ │ │ ├── LUFileParser.tokens │ │ │ │ │ │ ├── LUFileParserListener.js │ │ │ │ │ │ └── LUFileParserVisitor.js │ │ │ │ │ ├── importSection.js │ │ │ │ │ ├── luErrorListener.js │ │ │ │ │ ├── luParser.js │ │ │ │ │ ├── luResource.js │ │ │ │ │ ├── modelInfoSection.js │ │ │ │ │ ├── nestedIntentSection.js │ │ │ │ │ ├── newEntitySection.js │ │ │ │ │ ├── parseFileContents.js │ │ │ │ │ ├── qnaSection.js │ │ │ │ │ ├── read-text-file.js │ │ │ │ │ ├── sectionOperator.js │ │ │ │ │ ├── simpleIntentSection.js │ │ │ │ │ ├── translate-helpers.js │ │ │ │ │ └── visitor.js │ │ │ │ ├── luis/ │ │ │ │ │ ├── luConverter.js │ │ │ │ │ ├── luis.js │ │ │ │ │ ├── luisBuilder.js │ │ │ │ │ ├── luisCollate.js │ │ │ │ │ ├── luisGen.js │ │ │ │ │ ├── luisGenBuilder.js │ │ │ │ │ ├── luisValidator.js │ │ │ │ │ └── propertyHelper.js │ │ │ │ ├── qna/ │ │ │ │ │ ├── alterations/ │ │ │ │ │ │ ├── alterations.js │ │ │ │ │ │ └── qnaConverter.js │ │ │ │ │ └── qnamaker/ │ │ │ │ │ ├── kb.js │ │ │ │ │ ├── kbCollate.js │ │ │ │ │ ├── qnaContext.js │ │ │ │ │ ├── qnaConverter.js │ │ │ │ │ ├── qnaFiles.js │ │ │ │ │ ├── qnaList.js │ │ │ │ │ ├── qnaMakerBuilder.js │ │ │ │ │ ├── qnaMetaData.js │ │ │ │ │ ├── qnaPrompt.js │ │ │ │ │ └── qnamaker.js │ │ │ │ ├── qnabuild/ │ │ │ │ │ ├── builder.ts │ │ │ │ │ ├── core.ts │ │ │ │ │ ├── cross-trained-recognizer.ts │ │ │ │ │ ├── multi-language-recognizer.ts │ │ │ │ │ ├── recognizer.ts │ │ │ │ │ ├── serviceBase.js │ │ │ │ │ └── settings.ts │ │ │ │ ├── test/ │ │ │ │ │ └── testhelper.ts │ │ │ │ ├── translator/ │ │ │ │ │ └── lutranslate.js │ │ │ │ └── utils/ │ │ │ │ ├── enums/ │ │ │ │ │ ├── CLI-errors.js │ │ │ │ │ ├── invalidchars.js │ │ │ │ │ ├── luisEntityTypeNameMap.js │ │ │ │ │ ├── luisEntityTypes.js │ │ │ │ │ ├── luisbuiltintypes.js │ │ │ │ │ ├── luisobjenum.js │ │ │ │ │ ├── lusectiontypes.js │ │ │ │ │ ├── parsecommands.js │ │ │ │ │ ├── parserconsts.js │ │ │ │ │ └── recognizertypes.js │ │ │ │ ├── exception.js │ │ │ │ └── helpers.js │ │ │ └── utils/ │ │ │ ├── filehelper.ts │ │ │ └── textfilereader.ts │ │ ├── test/ │ │ │ ├── commands/ │ │ │ │ └── luis/ │ │ │ │ └── convert.test.js │ │ │ ├── fixtures/ │ │ │ │ ├── empty.json │ │ │ │ ├── empty.lu │ │ │ │ ├── es/ │ │ │ │ │ └── file.lu │ │ │ │ ├── examples/ │ │ │ │ │ ├── 1.lu │ │ │ │ │ ├── 10.lu │ │ │ │ │ ├── 11.lu │ │ │ │ │ ├── 12.lu │ │ │ │ │ ├── 13.lu │ │ │ │ │ ├── 2.lu │ │ │ │ │ ├── 3.lu │ │ │ │ │ ├── 4.lu │ │ │ │ │ ├── 5.lu │ │ │ │ │ ├── 6.lu │ │ │ │ │ ├── 7.lu │ │ │ │ │ ├── 8.lu │ │ │ │ │ ├── 9.lu │ │ │ │ │ ├── 9a.lu │ │ │ │ │ ├── README.MD │ │ │ │ │ ├── all.lu │ │ │ │ │ ├── all.md │ │ │ │ │ ├── all.qna │ │ │ │ │ ├── buyChocolate.lu │ │ │ │ │ ├── changeAlarm.lu │ │ │ │ │ ├── comment.lu │ │ │ │ │ ├── compositeEntities.lu │ │ │ │ │ ├── en-us/ │ │ │ │ │ │ ├── changeAlarm.lu │ │ │ │ │ │ ├── none.lu │ │ │ │ │ │ ├── qna4.lu │ │ │ │ │ │ └── root.lu │ │ │ │ │ ├── fr-fr/ │ │ │ │ │ │ ├── changeAlarm.lu │ │ │ │ │ │ ├── none.lu │ │ │ │ │ │ ├── qna4.lu │ │ │ │ │ │ └── root.lu │ │ │ │ │ ├── luFileReference1.lu │ │ │ │ │ ├── luFileReference2.lu │ │ │ │ │ ├── luFileReference3.lu │ │ │ │ │ ├── luFileReference4.lu │ │ │ │ │ ├── luFileReference5.lu │ │ │ │ │ ├── luFileReference6.lu │ │ │ │ │ ├── negativeCase1.lu │ │ │ │ │ ├── negativeCase2.lu │ │ │ │ │ ├── negativeCase3.lu │ │ │ │ │ ├── negativeCase4.lu │ │ │ │ │ ├── negativeCase5.lu │ │ │ │ │ ├── newEntityWithFeatures.lu │ │ │ │ │ ├── none.lu │ │ │ │ │ ├── patterns.lu │ │ │ │ │ ├── patterns1.lu │ │ │ │ │ ├── phraseLists.lu │ │ │ │ │ ├── qna-alterations.lu │ │ │ │ │ ├── qna1.lu │ │ │ │ │ ├── qna2.lu │ │ │ │ │ ├── qna3.lu │ │ │ │ │ ├── qna4.lu │ │ │ │ │ ├── qna5.lu │ │ │ │ │ ├── qna6.lu │ │ │ │ │ ├── qna7.lu │ │ │ │ │ ├── regexEntity.lu │ │ │ │ │ └── v7UpgradeTest.lu │ │ │ │ ├── file.lu │ │ │ │ ├── fr/ │ │ │ │ │ └── file.lu │ │ │ │ ├── generate/ │ │ │ │ │ ├── ClosedLists.cs │ │ │ │ │ ├── ClosedLists.json │ │ │ │ │ ├── CompositeEntities.cs │ │ │ │ │ ├── CompositeEntities.json │ │ │ │ │ ├── ContosoApp.cs │ │ │ │ │ ├── ContosoApp.json │ │ │ │ │ ├── FlightBooking.cs │ │ │ │ │ ├── FlightBooking.json │ │ │ │ │ ├── FlightBooking.ts │ │ │ │ │ ├── Intents.cs │ │ │ │ │ ├── Intents.json │ │ │ │ │ ├── NamespaceClass.cs │ │ │ │ │ ├── PatternEntities.cs │ │ │ │ │ ├── PatternEntities.json │ │ │ │ │ ├── PrebuiltEntities.cs │ │ │ │ │ ├── PrebuiltEntities.json │ │ │ │ │ ├── RegexEntities.cs │ │ │ │ │ ├── RegexEntities.json │ │ │ │ │ ├── SimpleEntities.cs │ │ │ │ │ ├── SimpleEntities.json │ │ │ │ │ ├── closed-lists.ts │ │ │ │ │ ├── composite-entities.ts │ │ │ │ │ ├── contoso-app.ts │ │ │ │ │ ├── intents.ts │ │ │ │ │ ├── invalid.json │ │ │ │ │ ├── pattern-entities.ts │ │ │ │ │ ├── prebuilt-entities.ts │ │ │ │ │ ├── regex-entities.ts │ │ │ │ │ └── simple-entities.ts │ │ │ │ ├── none.lu │ │ │ │ ├── qna.lu │ │ │ │ ├── root.luis.json │ │ │ │ ├── testcases/ │ │ │ │ │ ├── 1-intent-pattern-prebuilt.lu │ │ │ │ │ ├── 1-intent.lu │ │ │ │ │ ├── 8.lu │ │ │ │ │ ├── Child_Entity_With_Spaces.json │ │ │ │ │ ├── InvalidLUISModel.json │ │ │ │ │ ├── InvalidLUISModel1.json │ │ │ │ │ ├── InvalidLUISModel2.json │ │ │ │ │ ├── LUISAppWithPAInherits.json │ │ │ │ │ ├── ListEntityAndBatchtestsProblem.1.lu │ │ │ │ │ ├── ListEntityAndBatchtestsProblem.2.lu │ │ │ │ │ ├── ListEntityAndBatchtestsProblem.lu │ │ │ │ │ ├── MultiturnReplaceKbWithFlattenedTree.json │ │ │ │ │ ├── Skills/ │ │ │ │ │ │ ├── Calendar/ │ │ │ │ │ │ │ ├── Calendar.lu │ │ │ │ │ │ │ └── General.lu │ │ │ │ │ │ ├── Email/ │ │ │ │ │ │ │ ├── Email.lu │ │ │ │ │ │ │ └── General.lu │ │ │ │ │ │ └── Todo/ │ │ │ │ │ │ ├── General.lu │ │ │ │ │ │ └── ToDo.lu │ │ │ │ │ ├── all-entity-types.lu │ │ │ │ │ ├── all.json │ │ │ │ │ ├── all_qna.json │ │ │ │ │ ├── bad.lu │ │ │ │ │ ├── bad2.lu │ │ │ │ │ ├── bad3.lu │ │ │ │ │ ├── bad3a.lu │ │ │ │ │ ├── bad4.lu │ │ │ │ │ ├── bad5.lu │ │ │ │ │ ├── bf-627.json │ │ │ │ │ ├── buyChocolate.lu │ │ │ │ │ ├── calendar_all_prebuilt.json │ │ │ │ │ ├── collate/ │ │ │ │ │ │ ├── 1.lu │ │ │ │ │ │ ├── 10.lu │ │ │ │ │ │ ├── 11.lu │ │ │ │ │ │ ├── 11.qna │ │ │ │ │ │ ├── 12.lu │ │ │ │ │ │ ├── 12.qna │ │ │ │ │ │ ├── 2.lu │ │ │ │ │ │ ├── 3.lu │ │ │ │ │ │ ├── 4.lu │ │ │ │ │ │ ├── 5.lu │ │ │ │ │ │ ├── 6.lu │ │ │ │ │ │ ├── 7.lu │ │ │ │ │ │ ├── 8.lu │ │ │ │ │ │ ├── 9.lu │ │ │ │ │ │ ├── 9a.lu │ │ │ │ │ │ ├── buyChocolate.lu │ │ │ │ │ │ ├── changeAlarm.lu │ │ │ │ │ │ ├── comment.lu │ │ │ │ │ │ ├── none.lu │ │ │ │ │ │ ├── patterns1.lu │ │ │ │ │ │ ├── phraseLists.lu │ │ │ │ │ │ ├── qna-alterations.qna │ │ │ │ │ │ ├── qna1.qna │ │ │ │ │ │ ├── qna2.qna │ │ │ │ │ │ ├── qna3.qna │ │ │ │ │ │ ├── qna4.qna │ │ │ │ │ │ ├── qna5.qna │ │ │ │ │ │ ├── qna6.lu │ │ │ │ │ │ ├── qna6.qna │ │ │ │ │ │ └── qna7.qna │ │ │ │ │ ├── comment.lu │ │ │ │ │ ├── deepreference/ │ │ │ │ │ │ ├── qna1.lu │ │ │ │ │ │ └── qna5.lu │ │ │ │ │ ├── emptyIntentDescriptors.json │ │ │ │ │ ├── entityNameWithSpaceAndFeature.json │ │ │ │ │ ├── faq.lu │ │ │ │ │ ├── fileReference/ │ │ │ │ │ │ ├── 9.lu │ │ │ │ │ │ ├── AboutBot.lu │ │ │ │ │ │ └── L2/ │ │ │ │ │ │ ├── AboutUser.lu │ │ │ │ │ │ ├── Command.lu │ │ │ │ │ │ ├── Compliment.lu │ │ │ │ │ │ ├── Criticism.lu │ │ │ │ │ │ ├── Dialog.lu │ │ │ │ │ │ ├── Greeting.lu │ │ │ │ │ │ ├── Relationship.lu │ │ │ │ │ │ └── none.lu │ │ │ │ │ ├── import-resolver/ │ │ │ │ │ │ ├── lu-import-resolver/ │ │ │ │ │ │ │ ├── common.en-us.lu │ │ │ │ │ │ │ ├── help.en-us.lu │ │ │ │ │ │ │ ├── terminate.en-us.lu │ │ │ │ │ │ │ └── welcome.en-us.lu │ │ │ │ │ │ └── qna-import-resolver/ │ │ │ │ │ │ ├── common.en-us.qna │ │ │ │ │ │ ├── terminate.en-us.qna │ │ │ │ │ │ └── welcome.en-us.qna │ │ │ │ │ ├── intentWithSpace.json │ │ │ │ │ ├── invalid-alterations.lu │ │ │ │ │ ├── invalid-entity-definition.lu │ │ │ │ │ ├── invalid_model.lu │ │ │ │ │ ├── invalid_prebuilt_1.lu │ │ │ │ │ ├── invalid_prebuilt_2.lu │ │ │ │ │ ├── invalid_prebuilt_3.lu │ │ │ │ │ ├── invalid_prebuilt_4.lu │ │ │ │ │ ├── list-entity-body-without-hyphen.lu │ │ │ │ │ ├── lubuild/ │ │ │ │ │ │ ├── file-name-duplicated/ │ │ │ │ │ │ │ ├── Foo.en-us.lu │ │ │ │ │ │ │ └── Foo.lu │ │ │ │ │ │ ├── foo/ │ │ │ │ │ │ │ ├── config/ │ │ │ │ │ │ │ │ └── luis.settings.development.westus.json │ │ │ │ │ │ │ ├── dialogs/ │ │ │ │ │ │ │ │ ├── foo.en-us.lu.dialog │ │ │ │ │ │ │ │ ├── foo.fr-fr.lu.dialog │ │ │ │ │ │ │ │ ├── foo.lu.dialog │ │ │ │ │ │ │ │ └── foo.zh-cn.lu.dialog │ │ │ │ │ │ │ └── lufiles/ │ │ │ │ │ │ │ ├── foo.fr-fr.lu │ │ │ │ │ │ │ ├── foo.lu │ │ │ │ │ │ │ └── foo.zh-cn.lu │ │ │ │ │ │ ├── foo2/ │ │ │ │ │ │ │ ├── config/ │ │ │ │ │ │ │ │ └── luis.settings.development.westus.json │ │ │ │ │ │ │ ├── dialogs/ │ │ │ │ │ │ │ │ ├── foo.en-us.lu.dialog │ │ │ │ │ │ │ │ ├── foo.fr-fr.lu.dialog │ │ │ │ │ │ │ │ ├── foo.lu.dialog │ │ │ │ │ │ │ │ └── foo.zh-cn.lu.dialog │ │ │ │ │ │ │ ├── lufiles-and-dialog-assets/ │ │ │ │ │ │ │ │ ├── foo.en-us.lu.dialog │ │ │ │ │ │ │ │ ├── foo.fr-fr.lu │ │ │ │ │ │ │ │ ├── foo.fr-fr.lu.dialog │ │ │ │ │ │ │ │ ├── foo.lu │ │ │ │ │ │ │ │ ├── foo.lu.dialog │ │ │ │ │ │ │ │ ├── foo.zh-cn.lu │ │ │ │ │ │ │ │ └── luis.settings.development.westus.json │ │ │ │ │ │ │ └── luis/ │ │ │ │ │ │ │ ├── test(development)-foo.en-us.lu.json │ │ │ │ │ │ │ └── test(development)-foo.fr-fr.lu.json │ │ │ │ │ │ └── sandwich/ │ │ │ │ │ │ ├── config/ │ │ │ │ │ │ │ └── luis.settings.development.westus.json │ │ │ │ │ │ ├── dialogs/ │ │ │ │ │ │ │ ├── sandwich.en-us.lu.dialog │ │ │ │ │ │ │ └── sandwich.lu.dialog │ │ │ │ │ │ ├── lufiles/ │ │ │ │ │ │ │ ├── sandwich-BreadEntity.en-us.lu │ │ │ │ │ │ │ ├── sandwich-CheeseEntity.en-us.lu │ │ │ │ │ │ │ ├── sandwich-Confirmation.en-us.lu │ │ │ │ │ │ │ ├── sandwich-DirectionsEntity.en-us.lu │ │ │ │ │ │ │ ├── sandwich-LengthEntity.en-us.lu │ │ │ │ │ │ │ ├── sandwich-MeatEntity.en-us.lu │ │ │ │ │ │ │ ├── sandwich-NameEntity.en-us.lu │ │ │ │ │ │ │ ├── sandwich-PROPERTYName.en-us.lu │ │ │ │ │ │ │ ├── sandwich-Price.quantity.en-us.lu │ │ │ │ │ │ │ ├── sandwich-Quantity.en-us.lu │ │ │ │ │ │ │ ├── sandwich-QuantityEntity.en-us.lu │ │ │ │ │ │ │ ├── sandwich-cancelIntent.en-us.lu │ │ │ │ │ │ │ ├── sandwich-dimension.en-us.lu │ │ │ │ │ │ │ ├── sandwich-helpIntent.en-us.lu │ │ │ │ │ │ │ ├── sandwich-money.en-us.lu │ │ │ │ │ │ │ ├── sandwich-noneIntent.en-us.lu │ │ │ │ │ │ │ ├── sandwich-number.en-us.lu │ │ │ │ │ │ │ ├── sandwich-sandwich.lu │ │ │ │ │ │ │ └── sandwich.en-us.lu │ │ │ │ │ │ └── luis/ │ │ │ │ │ │ ├── test(development)-sandwich.en-us.lu.json │ │ │ │ │ │ ├── test(development)-sandwich.utteranceAdded.en-us.lu.json │ │ │ │ │ │ └── test(development)-sandwich.utteranceChanged.en-us.lu.json │ │ │ │ │ ├── merge_intents_disabled.lu │ │ │ │ │ ├── missing-synonyms.lu │ │ │ │ │ ├── missing-utterance.lu │ │ │ │ │ ├── missing-utterance2.lu │ │ │ │ │ ├── multi-ref.lu │ │ │ │ │ ├── multiturn.qna │ │ │ │ │ ├── nested-luis-json.json │ │ │ │ │ ├── newEntity1.json │ │ │ │ │ ├── newEntity2.json │ │ │ │ │ ├── overlappingEntities.json │ │ │ │ │ ├── phraseLists.lu │ │ │ │ │ ├── plFeatureDisabled.json │ │ │ │ │ ├── plWithFlags.lu │ │ │ │ │ ├── prebuilt-entity.lu │ │ │ │ │ ├── prebuilt_model.json │ │ │ │ │ ├── qna-alterations_Alterations.json │ │ │ │ │ ├── qna-filter-line-without-hyphen.lu │ │ │ │ │ ├── qna-question-line-without-hyphen.lu │ │ │ │ │ ├── qnaDocuments.json │ │ │ │ │ ├── qnaref.qna │ │ │ │ │ ├── reduced.lu │ │ │ │ │ ├── ref1.lu │ │ │ │ │ ├── ref10.lu │ │ │ │ │ ├── ref11.lu │ │ │ │ │ ├── ref12.lu │ │ │ │ │ ├── ref13.lu │ │ │ │ │ ├── ref14.qna │ │ │ │ │ ├── ref15.qna │ │ │ │ │ ├── ref2.lu │ │ │ │ │ ├── ref3.qna │ │ │ │ │ ├── ref4.lu │ │ │ │ │ ├── ref5.lu │ │ │ │ │ ├── ref6.lu │ │ │ │ │ ├── ref7.lu │ │ │ │ │ ├── ref8.lu │ │ │ │ │ ├── ref9.lu │ │ │ │ │ ├── ref_src1.lu │ │ │ │ │ ├── ref_src2.lu │ │ │ │ │ ├── regexmodel.luis │ │ │ │ │ ├── root.lu │ │ │ │ │ ├── root2.lu │ │ │ │ │ ├── root3.lu │ │ │ │ │ ├── section_disabled.lu │ │ │ │ │ ├── section_disabled2.lu │ │ │ │ │ ├── section_enabled.lu │ │ │ │ │ ├── section_enabled2.lu │ │ │ │ │ ├── special-char-in-entity-type.lu │ │ │ │ │ ├── test269-d.json │ │ │ │ │ ├── testTokenizerVersion.json │ │ │ │ │ ├── testcases-data.js │ │ │ │ │ ├── translate-testcase-data.js │ │ │ │ │ ├── translate-with-number.lu │ │ │ │ │ ├── utterance-without-hyphen.lu │ │ │ │ │ ├── v6WithoutPhraseLists.lu │ │ │ │ │ └── v7app.json │ │ │ │ ├── translation/ │ │ │ │ │ ├── en/ │ │ │ │ │ │ ├── qna.json │ │ │ │ │ │ ├── qna.lu │ │ │ │ │ │ └── translateLuResponse.json │ │ │ │ │ ├── files/ │ │ │ │ │ │ ├── allEntities.lu │ │ │ │ │ │ ├── bad.lu │ │ │ │ │ │ ├── fileRef.lu │ │ │ │ │ │ ├── intentsAndUtterances.lu │ │ │ │ │ │ ├── labelledEntityValue.lu │ │ │ │ │ │ ├── newEntity.lu │ │ │ │ │ │ ├── normalizedValue.lu │ │ │ │ │ │ ├── options.lu │ │ │ │ │ │ ├── phraseList.lu │ │ │ │ │ │ ├── qna.lu │ │ │ │ │ │ └── qnaContent.lu │ │ │ │ │ ├── fr/ │ │ │ │ │ │ ├── qna.json │ │ │ │ │ │ ├── qna.lu │ │ │ │ │ │ └── root.luis.json │ │ │ │ │ ├── serviceresponses/ │ │ │ │ │ │ ├── allEntities.json │ │ │ │ │ │ ├── fileRef.json │ │ │ │ │ │ ├── intentsAndUtterances.json │ │ │ │ │ │ ├── labelledEntityValue.json │ │ │ │ │ │ ├── newEntity.json │ │ │ │ │ │ ├── normalizedValue.json │ │ │ │ │ │ ├── options.json │ │ │ │ │ │ ├── phraseList.json │ │ │ │ │ │ ├── qna.json │ │ │ │ │ │ └── qnaContent.json │ │ │ │ │ ├── translateLuResponse.json │ │ │ │ │ ├── translateLuResponseSecond.json │ │ │ │ │ ├── translateLuisResponse.json │ │ │ │ │ ├── translateLuisResponseSecond.json │ │ │ │ │ └── translatedfiles/ │ │ │ │ │ ├── allEntities.lu │ │ │ │ │ ├── fileRef.lu │ │ │ │ │ ├── intentsAndUtterances.lu │ │ │ │ │ ├── labelledEntityValue.lu │ │ │ │ │ ├── newEntity.lu │ │ │ │ │ ├── normalizedValue.lu │ │ │ │ │ ├── options.lu │ │ │ │ │ ├── phraseList.lu │ │ │ │ │ ├── qna.lu │ │ │ │ │ └── qnaContent.lu │ │ │ │ └── verified/ │ │ │ │ ├── 1.json │ │ │ │ ├── 1.lu │ │ │ │ ├── 11.json │ │ │ │ ├── 12.json │ │ │ │ ├── 12qna.json │ │ │ │ ├── 13.json │ │ │ │ ├── 2.json │ │ │ │ ├── 3.json │ │ │ │ ├── 4.json │ │ │ │ ├── 5.json │ │ │ │ ├── 6.json │ │ │ │ ├── 7.json │ │ │ │ ├── 9.json │ │ │ │ ├── 9a.json │ │ │ │ ├── Child_Entity_With_Spaces.lu │ │ │ │ ├── LUISAppWithPAInherits.lu │ │ │ │ ├── LUISAppWithPAInherits.lu.json │ │ │ │ ├── ListEntityAndBatchtestsProblem.1_LUISBatchTest.json │ │ │ │ ├── ListEntityAndBatchtestsProblem.2_LUISBatchTest.json │ │ │ │ ├── ListEntityAndBatchtestsProblem_LUISBatchTest.json │ │ │ │ ├── MultiturnReplaceKbWithFlattenedTree.qna │ │ │ │ ├── Skills/ │ │ │ │ │ ├── Calendar.json │ │ │ │ │ ├── Email.json │ │ │ │ │ └── ToDo.json │ │ │ │ ├── all-entity-types.json │ │ │ │ ├── all-qna.json │ │ │ │ ├── all.json │ │ │ │ ├── allGen.lu │ │ │ │ ├── allGenQnA.lu │ │ │ │ ├── allRefresh.lu │ │ │ │ ├── allall-qna.lu │ │ │ │ ├── alterations_qna4.json │ │ │ │ ├── buyChocolate.json │ │ │ │ ├── calendar_all_prebuilt.lu │ │ │ │ ├── calendar_all_prebuilt_parsed.json │ │ │ │ ├── collate-qna.json │ │ │ │ ├── collate_Alterations.json │ │ │ │ ├── collate_alteration_gen.lu │ │ │ │ ├── collate_refresh.lu │ │ │ │ ├── collated-luis.json │ │ │ │ ├── collated-luis_LUISBatchTest.json │ │ │ │ ├── de/ │ │ │ │ │ ├── all.lu │ │ │ │ │ ├── faq.lu │ │ │ │ │ ├── reduced.lu │ │ │ │ │ └── translate-with-number.lu │ │ │ │ ├── emptyIntentDescriptors.lu │ │ │ │ ├── importUrl.json │ │ │ │ ├── luis_sorted.lu │ │ │ │ ├── merge_intents_disabled.json │ │ │ │ ├── modelAsFeatureGen.lu │ │ │ │ ├── modelAsFeatures.json │ │ │ │ ├── modelAsFeatures.lu │ │ │ │ ├── modelInfo.lu │ │ │ │ ├── multiturn.json.qna │ │ │ │ ├── multiturn.qna.json │ │ │ │ ├── nDepthEntity.json │ │ │ │ ├── nDepthEntity.lu │ │ │ │ ├── nDepthEntityInUtterance.json │ │ │ │ ├── nDepthEntityInUtterance.lu │ │ │ │ ├── newEntity1.lu │ │ │ │ ├── newEntity2.lu │ │ │ │ ├── newEntityWithFeatures.json │ │ │ │ ├── newEntityWithFeatures.lu │ │ │ │ ├── plFeatures.json │ │ │ │ ├── plFeatures.lu │ │ │ │ ├── plWithFlags.json │ │ │ │ ├── prebuilt-entity.json │ │ │ │ ├── prebuilt_mode.lu │ │ │ │ ├── prebuilt_model_parse.json │ │ │ │ ├── qna5.json │ │ │ │ ├── qna_a_sorted.lu │ │ │ │ ├── qna_sorted.lu │ │ │ │ ├── ref1.json │ │ │ │ ├── ref14_a.json │ │ │ │ ├── ref15_a.json │ │ │ │ ├── ref4.json │ │ │ │ ├── ref5.json │ │ │ │ ├── ref6.json │ │ │ │ ├── ref7.json │ │ │ │ ├── ref8.json │ │ │ │ ├── referenceUrl.json │ │ │ │ ├── referenceUrlWithWildCard.json │ │ │ │ ├── regexmodel.lu │ │ │ │ ├── root.json │ │ │ │ ├── root2.json │ │ │ │ ├── root2_luis.json │ │ │ │ ├── root3.lu │ │ │ │ ├── root_luis.json │ │ │ │ ├── section_disabled.json │ │ │ │ ├── section_disabled2.json │ │ │ │ ├── section_enabled.json │ │ │ │ ├── section_enabled2.json │ │ │ │ ├── sorted.lu │ │ │ │ ├── special-char-in-entity-type.json │ │ │ │ ├── stdin-qna.lu │ │ │ │ ├── stdin.lu │ │ │ │ ├── test269-d.lu │ │ │ │ ├── v5Upgrade.json │ │ │ │ ├── v5UpgradeTest.lu │ │ │ │ ├── v6WithoutPhraseLists.json │ │ │ │ ├── v7UpgradeTest.json │ │ │ │ ├── v7UpgradeTest.lu │ │ │ │ ├── v7app.lu │ │ │ │ ├── v7app_c.json │ │ │ │ └── zh-Hans/ │ │ │ │ └── reduced.lu │ │ │ ├── hook.js │ │ │ ├── mocha.opts │ │ │ ├── parser/ │ │ │ │ ├── cross-train/ │ │ │ │ │ └── crossTrainer.test.js │ │ │ │ ├── lu/ │ │ │ │ │ └── lu.test.js │ │ │ │ ├── lubuild/ │ │ │ │ │ └── lubuild.test.js │ │ │ │ ├── lufile/ │ │ │ │ │ ├── appKbMetaData.test.js │ │ │ │ │ ├── classes/ │ │ │ │ │ │ └── classes.test.js │ │ │ │ │ ├── helpers.test.js │ │ │ │ │ ├── luQnAFileReference.test.js │ │ │ │ │ ├── luis.boundary.test.js │ │ │ │ │ ├── lutranslator.test.js │ │ │ │ │ ├── newEntityDefinitionWithSpaces.test.js │ │ │ │ │ ├── parseFileContents.NewEntityDefinition.test.js │ │ │ │ │ ├── parseFileContents.comments.test.js │ │ │ │ │ ├── parseFileContents.composeEntity.test.js │ │ │ │ │ ├── parseFileContents.featureToModels.test.js │ │ │ │ │ ├── parseFileContents.modelAsFeature.test.js │ │ │ │ │ ├── parseFileContents.nDepthEntity.test.js │ │ │ │ │ ├── parseFileContents.newEntityInUtterance.test.js │ │ │ │ │ ├── parseFileContents.parseFile.test.js │ │ │ │ │ ├── parseFileContents.qnafile.test.js │ │ │ │ │ ├── parseFileContents.regexEntity.test.js │ │ │ │ │ ├── parseFileContents.roleTest.test.js │ │ │ │ │ ├── qnaMultiTurnAndReferences.test.js │ │ │ │ │ ├── sectionapi.test.js │ │ │ │ │ └── urlReference.test.js │ │ │ │ ├── luis/ │ │ │ │ │ └── luisBuilder.test.js │ │ │ │ ├── qna/ │ │ │ │ │ └── qnaMakerBuilder.test.js │ │ │ │ └── qnabuild/ │ │ │ │ └── qnabuild.test.js │ │ │ ├── tsconfig.json │ │ │ └── utils/ │ │ │ └── filehelper.test.js │ │ ├── tsconfig.json │ │ └── tslint.json │ ├── rush.json │ └── scripts/ │ └── fixurls.js ├── schemas/ │ ├── component/ │ │ ├── component.schema │ │ ├── definitions.schema │ │ ├── readme.md │ │ └── v1.0/ │ │ └── component.schema │ ├── protocol/ │ │ └── botframework.json │ ├── readme.md │ ├── skills/ │ │ ├── SchemaManifestTests/ │ │ │ ├── SchemaManifestTests.csproj │ │ │ ├── SchemaManifestTests.sln │ │ │ └── ValidateSchemaTests.cs │ │ ├── readme.md │ │ ├── skill-manifest-2.0.0.json │ │ ├── skill-manifest-2.1.preview-1.json │ │ ├── v2.0/ │ │ │ ├── samples/ │ │ │ │ ├── complex-skillmanifest.json │ │ │ │ ├── echo-skillmanifest.json │ │ │ │ └── simple-skillmanifest.json │ │ │ └── skill-manifest.json │ │ ├── v2.1/ │ │ │ ├── samples/ │ │ │ │ ├── complex-pva-manifest.json │ │ │ │ ├── complex-skillmanifest.json │ │ │ │ ├── echo-skillmanifest.json │ │ │ │ └── simple-skillmanifest.json │ │ │ └── skill-manifest.json │ │ ├── v2.1.preview-1/ │ │ │ ├── samples/ │ │ │ │ ├── complex-skillmanifest.json │ │ │ │ ├── echo-skillmanifest.json │ │ │ │ └── simple-skillmanifest.json │ │ │ └── skill-manifest.json │ │ └── v2.2/ │ │ ├── samples/ │ │ │ ├── complex-pva-manifest.json │ │ │ ├── complex-skillmanifest.json │ │ │ ├── echo-skillmanifest.json │ │ │ ├── relativeUris/ │ │ │ │ ├── complex-skillmanifest.json │ │ │ │ ├── knowledge-base/ │ │ │ │ │ ├── SkillBot-QnA-en.qna │ │ │ │ │ ├── SkillBot-QnA-es-ES.qna │ │ │ │ │ └── SkillBot-QnA-es-MX.qna │ │ │ │ ├── language-understanding/ │ │ │ │ │ ├── SkillBot-en.lu │ │ │ │ │ ├── SkillBot-es-ES.lu │ │ │ │ │ └── SkillBot-es-MX.lu │ │ │ │ └── privacy.html │ │ │ └── simple-skillmanifest.json │ │ └── skill-manifest.json │ └── ui/ │ └── v1.0/ │ └── ui.schema ├── specs/ │ ├── botframework-activity/ │ │ ├── botframework-activity.md │ │ └── botframework-cards.md │ ├── botframework-protocol/ │ │ ├── botframework-channel.json │ │ ├── channel-update-spec.md │ │ ├── directline-1.1.json │ │ └── directline-3.0.json │ ├── component-model/ │ │ └── overview.md │ ├── manifest/ │ │ ├── botframework-manifest.md │ │ ├── example.man │ │ └── manifest.json │ ├── testing/ │ │ ├── skills/ │ │ │ ├── SkillsFunctionalTesting.md │ │ │ ├── SkillsMatrixTesting.md │ │ │ ├── SkillsScopeInformation.md │ │ │ ├── media/ │ │ │ │ └── src/ │ │ │ │ └── SkillsFunctionalTestingDiagrams.pptx │ │ │ └── scenarios/ │ │ │ ├── AuthWithOAuthCard.md │ │ │ ├── AuthWithSSO.md │ │ │ ├── CardActionsWithInvokes.md │ │ │ ├── DraftScenarios.md │ │ │ ├── MultiTurnInteraction.md │ │ │ ├── ProactiveMessage.md │ │ │ ├── SingleTurnInteraction.md │ │ │ ├── SkillCallsSkill.md │ │ │ ├── SkillReceivesAttachment.md │ │ │ ├── SkillStartsConversation.md │ │ │ ├── TeamsAPI.md │ │ │ ├── TeamsSkillWithTaskModule.md │ │ │ └── UpdateDeleteAdaptiveCard.md │ │ └── testing.md │ └── transcript/ │ └── transcript.md └── tools/ └── changelog/ ├── .gitignore └── git-log.js ================================================ FILE CONTENTS ================================================ ================================================ FILE: .editorconfig ================================================ root = true [*] insert_final_newline = true [*.json] indent_style = space indent_size = 4 [*.schema] indent_style = space indent_size = 4 ================================================ FILE: .gitattributes ================================================ ############################################################################### # Set default behavior to automatically normalize line endings. ############################################################################### * text=crlf ############################################################################### # Set default behavior for command prompt diff. # # This is need for earlier builds of msysgit that does not have it on by # default for csharp files. # Note: This is only used by command line ############################################################################### #*.cs diff=csharp ############################################################################### # Set the merge driver for project and solution files # # Merging from the command prompt will add diff markers to the files if there # are conflicts (Merging from VS is not affected by the settings below, in VS # the diff markers are never inserted). Diff markers may cause the following # file extensions to fail to load in VS. An alternative would be to treat # these files as binary and thus will always conflict and require user # intervention with every merge. To do so, just uncomment the entries below ############################################################################### #*.sln merge=binary #*.csproj merge=binary #*.vbproj merge=binary #*.vcxproj merge=binary #*.vcproj merge=binary #*.dbproj merge=binary #*.fsproj merge=binary #*.lsproj merge=binary #*.wixproj merge=binary #*.modelproj merge=binary #*.sqlproj merge=binary #*.wwaproj merge=binary ############################################################################### # behavior for image files # # image files are treated as binary by default. ############################################################################### #*.jpg binary #*.png binary #*.gif binary ############################################################################### # diff behavior for common document formats # # Convert binary document formats to text before diffing them. This feature # is only available from the command line. Turn it on by uncommenting the # entries below. ############################################################################### #*.doc diff=astextplain #*.DOC diff=astextplain #*.docx diff=astextplain #*.DOCX diff=astextplain #*.dot diff=astextplain #*.DOT diff=astextplain #*.pdf diff=astextplain #*.PDF diff=astextplain #*.rtf diff=astextplain #*.RTF diff=astextplain ================================================ FILE: .github/CODEOWNERS ================================================ # Lines starting with '#' are comments. # Each line is a file pattern followed by one or more owners. # More details are here: https://help.github.com/articles/about-codeowners/ # The '*' pattern is global owners. # Order is important. The last matching pattern has the most precedence. # The folders are ordered as follows: # In each subsection folders are ordered first by depth, then alphabetically. # This should make it easy to add new rules without breaking existing ones. # Global rule: * @microsoft/botframework-sdk ================================================ FILE: .github/ISSUE_TEMPLATE/bot-framework-bug.md ================================================ --- name: Bug about: Report a new Bot Framework bug. title: "" labels: "needs-triage, bug" assignees: "" --- ### [Github issues](https://github.com/microsoft/botframework-sdk) should be used for bugs and feature requests. Use [Stack Overflow](https://stackoverflow.com/questions/tagged/botframework) for general "how-to" questions. ## Version What package version of the SDK are you using. ## Describe the bug Give a clear and concise description of what the bug is. ## To Reproduce Steps to reproduce the behavior: 1. Go to '...' 2. Click on '....' 3. Scroll down to '....' 4. See error ## Expected behavior Give a clear and concise description of what you expected to happen. ## Screenshots If applicable, add screenshots to help explain your problem. ## Additional context Add any other context about the problem here. ## Tracking Status **Dotnet SDK** [TODO]() - [ ] PR - [ ] Merged **Javascript SDK** [TODO]() - [ ] PR - [ ] Merged **Python SDK** [TODO]() - [ ] PR - [ ] Merged **Java SDK** [TODO]() - [ ] PR - [ ] Merged **Samples** [TODO]() - [ ] PR - [ ] Merged **Docs** [TODO]() - [ ] PR - [ ] Merged **Tools** [TODO]() - [ ] PR - [ ] Merged ================================================ FILE: .github/ISSUE_TEMPLATE/bot-framework-feature-request.md ================================================ --- name: Feature Request about: Create a new Bot Framework feature request. title: "" labels: "needs-triage, feature-request" assignees: "" --- ## This is not for asking questions or filing bugs To get help with an issue with your bot * ["How To"](https://stackoverflow.com/help/how-to-ask) questions should be submitted on [Stack Overflow](https://stackoverflow.com/questions/tagged/botframework). To file an issue against a component please go to the components repo * [Create Azure Bot Framework Service or Channel issue](https://github.com/Microsoft/botframework-services/issues/new/choose) * [Create Bot Builder Tools issue](https://github.com/Microsoft/botbuilder-tools/issues/new/choose) * [Create Bot Builder SDK V3 (.NET or Javascript) issue](https://github.com/Microsoft/botbuilder-V3/issues/new/choose) * [Create Bot Builder SDK V4 .NET issue](https://github.com/Microsoft/botbuilder-dotnet/issues/new/choose) * [Create Bot Builder SDK V4 Javascript issue](https://github.com/Microsoft/botbuilder-js/issues/new/choose) * [Create Bot Builder SDK V4 Java issue](https://github.com/Microsoft/botbuilder-java/issues/new/choose) * [Create Bot Builder SDK V4 Python issue](https://github.com/Microsoft/botbuilder-python/issues/new/choose) ## Issue *Describe the issue you are addressing* ## Proposed change *Describe the proposed solution* ## Component Impact *Describe which components need to be updated* ## Customer Impact *Describe the impact on customers* ## Tracking Status **Dotnet SDK** [TODO]() - [ ] PR - [ ] Merged **Javascript SDK** [TODO]() - [ ] PR - [ ] Merged **Python SDK** [TODO]() - [ ] PR - [ ] Merged **Java SDK** [TODO]() - [ ] PR - [ ] Merged **Samples** [TODO]() - [ ] PR - [ ] Merged **Docs** [TODO]() - [ ] PR - [ ] Merged **Tools** [TODO]() - [ ] PR - [ ] Merged ================================================ FILE: .github/PULL_REQUEST_TEMPLATE.md ================================================ Fixes # ## Proposed Changes - - - ## Testing ================================================ FILE: .github/issue_template.md ================================================ ## This is not for asking questions or filing bugs To get help with an issue with your bot * ["How To"](https://stackoverflow.com/help/how-to-ask) questions should be submitted on [Stack Overflow](https://stackoverflow.com/questions/tagged/botframework) where there is a robust community to help you To file an issue against a component please go to the components repo * [Create Azure Bot Framework Service or Channel issue](https://github.com/Microsoft/botframework-services/issues/new/choose) * [Create Bot Builder Tools issue](https://github.com/Microsoft/botbuilder-tools/issues/new/choose) * [Create Bot Builder SDK V3 (.NET or Javascript) issue](https://github.com/Microsoft/botbuilder-V3/issues/new/choose) * [Create Bot Builder SDK V4 .NET issue](https://github.com/Microsoft/botbuilder-dotnet/issues/new/choose) * [Create Bot Builder SDK V4 Javascript issue](https://github.com/Microsoft/botbuilder-js/issues/new/choose) * [Create Bot Builder SDK V4 Java issue](https://github.com/Microsoft/botbuilder-java/issues/new/choose) * [Create Bot Builder SDK V4 Python issue](https://github.com/Microsoft/botbuilder-python/issues/new/choose) ================================================ FILE: .gitignore ================================================ ## Ignore Visual Studio temporary files, build results, and ## files generated by popular Visual Studio add-ons. # User-specific files *.suo *.user *.userosscache *.sln.docstates # User-specific files (MonoDevelop/Xamarin Studio) *.userprefs # Build results [Dd]ebug/ [Dd]ebugPublic/ [Rr]elease/ [Rr]eleases/ x64/ x86/ bld/ [Bb]in/ [Oo]bj/ ecf/ rcf/ bones/ .vs/ .vscode/ .config/ # Python virtual environments **/*env/ ================================================ FILE: CODE_OF_CONDUCT.md ================================================ # Microsoft Open Source Code of Conduct This project has adopted the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/). Resources: - [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/) - [Microsoft Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/) - Contact [opencode@microsoft.com](mailto:opencode@microsoft.com) with questions or concerns ================================================ FILE: Common/Telemetry/README.md ================================================ # Telemetry Analytics Bot Analytics provides telemetry for analyzing conversation activities. ## Requirements To enable bot analytics: * [Add Telemetry](https://aka.ms/AddBotTelemetry) via Application Insights in the Bot Code * [Enable Application Insights](https://docs.microsoft.com/en-us/azure/bot-service/bot-service-resources-app-insights-keys?view=azure-bot-service-4.0) on Azure Bot Service Once active, basic visualization is available out of the box on Azure Bot Service. In addition, one can deploy and customize more advanced analytics views. ## Conversation Health Dashboard The Conversation Health Dashboard template presents analytics view of bot conversation activity. ### Instructions ## See Also * [Analyze Bot Telemetry](https://aka.ms/AnalyzeBotTelemetry) : Learn how to analyze Application Insights log analytics data. * See [Azure ARM templates](https://docs.microsoft.com/en-us/azure/azure-resource-manager/templates/quickstart-create-templates-use-the-portal) Documentation ================================================ FILE: Common/Transcripts/CoreExtensionsTests/ConversationStateTest.chat ================================================ bot=bot user=user user:start test bot:bot message user:set foo user:read bot:value:foo user:delete user:read bot:value: user:set bar user:set test user:read bot:value:test user:read bot:value:test user:delete user:read bot:value: ================================================ FILE: Common/Transcripts/CoreExtensionsTests/CustomStateTest.chat ================================================ bot=bot user=user user:start test bot:bot message user:set foo user:read bot:value:foo user:delete user:read bot:value: user:set bar user:set test user:read bot:value:test user:read bot:value:test user:delete user:read bot:value: ================================================ FILE: Common/Transcripts/CoreExtensionsTests/UserStateTest.chat ================================================ bot=bot user=user user:start test bot:bot message user:set foo user:read bot:value:foo user:delete user:read bot:value: user:set bar user:set test user:read bot:value:test user:read bot:value:test user:delete user:read bot:value: ================================================ FILE: Common/Transcripts/CoreTests/BotAdapted_Bracketing.chat ================================================ bot=bot user=user bot:before message bot:conversationUpdate bot:after message user:use middleware bot:before message bot:using middleware bot:after message user:catch exception bot:before message bot:generating exception bot:Caught: exception to catch ================================================ FILE: Common/Transcripts/DialogsTests/AttachmentPrompt.chat ================================================ user=User bot=Bot User: hello Bot: please add an attachment. User: [Attachment=ContentAttachment.json] Bot: some content ================================================ FILE: Common/Transcripts/DialogsTests/ChoicePrompt.chat ================================================ user=user bot=bot user: Hello! bot: favorite color? (1) red, (2) green, or (3) blue user: red bot: Bot received the choice 'red'. user: Hello! bot: favorite color? (1) red, (2) green, or (3) blue user: 3 bot: Bot received the choice 'blue'. user: Hello! bot: favorite color? (1) red, (2) green, or (3) blue user: cat bot: I didn't catch that. Select a color from the list. (1) red, (2) green, or (3) blue user: green bot: Bot received the choice 'green'. ================================================ FILE: Common/Transcripts/DialogsTests/ConfirmPrompt.chat ================================================ user=User bot=Bot User: hello Bot: Please confirm. User: hello Bot: Please confirm, say 'yes' or 'no' or something like that. User: no Bot: Not confirmed. User: hello Bot: Please confirm. User: yes Bot: Confirmed. ================================================ FILE: Common/Transcripts/DialogsTests/ContentAttachment.json ================================================ "some content" ================================================ FILE: Common/Transcripts/DialogsTests/DateTimePrompt.chat ================================================ user=User bot=Bot User: hello Bot: What date would you like? User: cat Bot: Sorry, but that is not a date. What date would you like? User: 31 February 2018 Bot: Timex:'2018-02-31' Value:'not resolved' User: hello Bot: What date would you like? User: 5th December 2018 at 9am Bot: Timex:'2018-12-05T09' Value:'2018-12-05 09:00:00' ================================================ FILE: Common/Transcripts/DialogsTests/NumberPrompt.chat ================================================ user=User bot=Bot User: hello Bot: Enter a number. User: hello Bot: You must enter a valid positive number less than 100. User: 150 Bot: You must enter a valid positive number less than 100. User: 64 Bot: Bot received the number '64'. User: hello Bot: Enter a number. User: One Bot: Bot received the number '1'. ================================================ FILE: Common/Transcripts/DialogsTests/TextPrompt.chat ================================================ user=User bot=Bot User: hello Bot: Enter some text. User: hi Bot: Make sure the text is greater than three characters. User: hello Bot: Bot received the text 'hello'. ================================================ FILE: Common/Transcripts/DialogsTests/Waterfall.chat ================================================ user=User bot=Bot User: hello Bot: step1 User: hello Bot: step2 User: hello Bot: step3 ================================================ FILE: Common/Transcripts/DialogsTests/WaterfallNested.chat ================================================ user=User bot=Bot User: hello Bot: step1 Bot: step1.1 User: hello Bot: step1.2 User: hello Bot: step2 Bot: step2.1 User: hello Bot: step2.2 ================================================ FILE: Common/Transcripts/DialogsTests/WaterfallPrompt.chat ================================================ user=User bot=Bot User: hello Bot: step1 Bot: Enter a number. User: hello again Bot: It must be a number User: 42 Bot: Thanks for '42' Bot: step2 Bot: Enter a number. User: apple Bot: It must be a number User: orange Bot: It must be a number User: 64 Bot: Thanks for '64' Bot: step3 ================================================ FILE: Common/Transcripts/LuisTests/LuisMiddleware.chat ================================================ user=User bot=Bot user:test bot:default message user:add imax theater to my upcoming events bot:intent:Calendar_Add user:display weekend plans bot:intent:Calendar_Find user:test bot:default message ================================================ FILE: Common/Transcripts/QnATests/QnAMiddleware.chat ================================================ user=User bot=Bot user:random text bot:default message user:hi bot:hello user:test bot:message user:tests bot:message user:custom random text bot:default message ================================================ FILE: Common/Transcripts/README.md ================================================ # Bot Builder SDK v4 - Transcript Files This folder contains the transcript files (.chat files) used in BotBuilder-v4 for testing several components and guarantee the same behavior between the different platform (.NET, NodeJS, Python, Java). ## BotBuilder Developer writing new transcript-based tests 1. Clone/Fork the [BotBuilder repository](https://github.com/Microsoft/BotBuilder). 2. Create a branch and place the new transcripts in the [Common\Transcripts folder](https://github.com/southworkscom/BotBuilder/tree/botbuilder-v4-transcripts/Common/Transcripts) (this can be done locally without actually pushing changes at this stage) 3. Write code and create new tests. Set the `BOTBUILDER_TRANSCRIPTS_LOCATION` Environment var pointing to the local repo / transcript folder. E.g.: ```shell # On *nix: export BOTBUILDER_TRANSCRIPTS_LOCATION=~/projects/BotBuilder/Common/Transcripts ``` ```Batchfile REM On Windows: SET BOTBUILDER_TRANSCRIPTS_LOCATION=C:\Projects\BotBuilder\Common\Transcripts ``` 4. Run the transcript tests from the platform repository. The code will look into the BOTBUILDER_TRANSCRIPTS_LOCATION folder for all files. E.g.: ``` cd transcripts npm run test ``` ## Pushing changes to BotBuilder Repos 1. Create a PR to the [BotBuilder repository](https://github.com/Microsoft/BotBuilder) with the new transcripts. 2. Once the transcripts PR is merged, create a PR to the BotBuilder's platform repository with the code updates and tests. ================================================ FILE: Common/Transcripts/TranslationTests/LocaleConvertToEnglish.chat ================================================ user=User bot=Bot user:set language fr-ca user:Set a meeting on 2018-10-04 bot:message: Set a meeting on 10/4/2018 user:set language fr-fr user:Set a meeting on 26/08/2018 bot:message: Set a meeting on 8/26/2018 user:set language en-us user:Set a meeting on 05/29/2018 bot:message: Set a meeting on 05/29/2018 ================================================ FILE: Common/Transcripts/TranslationTests/TranslateToEnglish.chat ================================================ user=User bot=Bot user:set language es-es user:Hola bot:message: Hello user:set language fr-fr user:Salut bot:message: Hello user:set language en-us user:Hello bot:message: Hello ================================================ FILE: Common/Transcripts/TranslationTests/TranslateToUserLanguage.chat ================================================ user=User bot=Bot user:set language es-es user:Hola bot:mensaje: Hola user:set language fr-fr user:Salut bot:message: Bonjour user:set language en-us user:Hello bot:message: Hello ================================================ FILE: Common/badges/BotBuilderBadge.md ================================================ [![BotBuilder]( https://img.shields.io/badge/Powered%20By-BotBuilder-blue.svg?colorA=black&label=Powered%20By&logo=data%3Aimage%2Fsvg%2Bxml%3Bbase64%2CPD94bWwgdmVyc2lvbj0iMS4wIiBlbmNvZGluZz0idXRmLTgiPz48c3ZnIGlkPSJtYXN0ZXItYXJ0Ym9hcmQiIHZpZXdCb3g9IjAgMCAxNDAwIDk4MCIgdmVyc2lvbj0iMS4xIiB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHg9IjBweCIgeT0iMHB4IiBzdHlsZT0iZW5hYmxlLWJhY2tncm91bmQ6bmV3IDAgMCAxNDAwIDk4MDsiIHdpZHRoPSIxNDAwcHgiIGhlaWdodD0iOTgwcHgiPjxnIHRyYW5zZm9ybT0ibWF0cml4KDYxLjI1MDAwMzgxNDY5NzI2NiwgMCwgMCwgNjEuMjUwMDAzODE0Njk3MjY2LCAyMDkuOTk5OTY4NDY0NjY0LCAtMC4wMDAwMDEwMTc3NTc5OTUyMTQxOTY5KSI%2BPHRpdGxlPmljX2JvdF9mcmFtZXdvcms8L3RpdGxlPjxnIGlkPSJMYXllcl8yIiBkYXRhLW5hbWU9IkxheWVyIDIiPjxnIGlkPSJMYXllcl8xLTIiIGRhdGEtbmFtZT0iTGF5ZXIgMSI%2BPHBhdGggZD0iTSA3LjI4IDggQSAwLjczIDAuNzMgMCAwIDEgNi41NSA4LjczIEEgMC43MyAwLjczIDAgMCAxIDUuODIgOCBBIDAuNzMgMC43MyAwIDAgMSA2LjU1IDcuMjcgQSAwLjczIDAuNzMgMCAwIDEgNy4yOCA4IFoiIHN0eWxlPSJmaWxsOiByZ2IoMjU1LCAyNTUsIDI1NSk7Ii8%2BPHBhdGggZD0iTSAxMC4xOCA4IEEgMC43MyAwLjczIDAgMCAxIDkuNDUgOC43MyBBIDAuNzMgMC43MyAwIDAgMSA4LjcyIDggQSAwLjczIDAuNzMgMCAwIDEgOS40NSA3LjI3IEEgMC43MyAwLjczIDAgMCAxIDEwLjE4IDggWiIgc3R5bGU9ImZpbGw6IHJnYigyNTUsIDI1NSwgMjU1KTsiLz48cGF0aCBkPSJNIDYuOTcwMDAwMjY3MDI4ODA5IDExLjc3MDAwMDQ1Nzc2MzY3MiBMIDMuMjAwMDAwMDQ3NjgzNzE2IDggTCA2Ljk3MDAwMDI2NzAyODgwOSA0LjIzMDAwMDAxOTA3MzQ4NiBMIDYuMjg5OTk5OTYxODUzMDI3IDMuNTM5OTk5OTYxODUzMDI3MyBMIDEuODMwMDAwMDQyOTE1MzQ0MiA4IEwgNi4yODk5OTk5NjE4NTMwMjcgMTIuNDYwMDAwMDM4MTQ2OTczIEwgNi45NzAwMDAyNjcwMjg4MDkgMTEuNzcwMDAwNDU3NzYzNjcyIFoiIHN0eWxlPSJmaWxsOiByZ2IoMjU1LCAyNTUsIDI1NSk7Ii8%2BPHBhdGggZD0iTSA5LjAyOTk5OTczMjk3MTE5MSAxMS43NzAwMDA0NTc3NjM2NzIgTCA5LjcxMDAwMDAzODE0Njk3MyAxMi40NjAwMDAwMzgxNDY5NzMgTCAxNC4xNzAwMDAwNzYyOTM5NDUgOCBMIDkuNzEwMDAwMDM4MTQ2OTczIDMuNTM5OTk5OTYxODUzMDI3MyBMIDkuMDI5OTk5NzMyOTcxMTkxIDQuMjMwMDAwMDE5MDczNDg2IEwgMTIuODAwMDAwMTkwNzM0ODYzIDggTCA5LjAyOTk5OTczMjk3MTE5MSAxMS43NzAwMDA0NTc3NjM2NzIgWiIgc3R5bGU9ImZpbGw6IHJnYigyNTUsIDI1NSwgMjU1KTsiLz48cGF0aCBkPSJNOCwwYTgsOCwwLDEsMCw4LDhBOCw4LDAsMCwwLDgsMFpNOCwxNWE3LDcsMCwxLDEsNy03QTcsNywwLDAsMSw4LDE1WiIgc3R5bGU9ImZpbGw6IHJnYigyNTUsIDI1NSwgMjU1KTsiLz48L2c%2BPC9nPjwvZz48L3N2Zz4%3D)](https://github.com/Microsoft/BotBuilder) Markdown ``` [![BotBuilder]( https://img.shields.io/badge/Powered%20By-BotBuilder-blue.svg?colorA=black&label=Powered%20By&logo=data%3Aimage%2Fsvg%2Bxml%3Bbase64%2CPD94bWwgdmVyc2lvbj0iMS4wIiBlbmNvZGluZz0idXRmLTgiPz48c3ZnIGlkPSJtYXN0ZXItYXJ0Ym9hcmQiIHZpZXdCb3g9IjAgMCAxNDAwIDk4MCIgdmVyc2lvbj0iMS4xIiB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHg9IjBweCIgeT0iMHB4IiBzdHlsZT0iZW5hYmxlLWJhY2tncm91bmQ6bmV3IDAgMCAxNDAwIDk4MDsiIHdpZHRoPSIxNDAwcHgiIGhlaWdodD0iOTgwcHgiPjxnIHRyYW5zZm9ybT0ibWF0cml4KDYxLjI1MDAwMzgxNDY5NzI2NiwgMCwgMCwgNjEuMjUwMDAzODE0Njk3MjY2LCAyMDkuOTk5OTY4NDY0NjY0LCAtMC4wMDAwMDEwMTc3NTc5OTUyMTQxOTY5KSI%2BPHRpdGxlPmljX2JvdF9mcmFtZXdvcms8L3RpdGxlPjxnIGlkPSJMYXllcl8yIiBkYXRhLW5hbWU9IkxheWVyIDIiPjxnIGlkPSJMYXllcl8xLTIiIGRhdGEtbmFtZT0iTGF5ZXIgMSI%2BPHBhdGggZD0iTSA3LjI4IDggQSAwLjczIDAuNzMgMCAwIDEgNi41NSA4LjczIEEgMC43MyAwLjczIDAgMCAxIDUuODIgOCBBIDAuNzMgMC43MyAwIDAgMSA2LjU1IDcuMjcgQSAwLjczIDAuNzMgMCAwIDEgNy4yOCA4IFoiIHN0eWxlPSJmaWxsOiByZ2IoMjU1LCAyNTUsIDI1NSk7Ii8%2BPHBhdGggZD0iTSAxMC4xOCA4IEEgMC43MyAwLjczIDAgMCAxIDkuNDUgOC43MyBBIDAuNzMgMC43MyAwIDAgMSA4LjcyIDggQSAwLjczIDAuNzMgMCAwIDEgOS40NSA3LjI3IEEgMC43MyAwLjczIDAgMCAxIDEwLjE4IDggWiIgc3R5bGU9ImZpbGw6IHJnYigyNTUsIDI1NSwgMjU1KTsiLz48cGF0aCBkPSJNIDYuOTcwMDAwMjY3MDI4ODA5IDExLjc3MDAwMDQ1Nzc2MzY3MiBMIDMuMjAwMDAwMDQ3NjgzNzE2IDggTCA2Ljk3MDAwMDI2NzAyODgwOSA0LjIzMDAwMDAxOTA3MzQ4NiBMIDYuMjg5OTk5OTYxODUzMDI3IDMuNTM5OTk5OTYxODUzMDI3MyBMIDEuODMwMDAwMDQyOTE1MzQ0MiA4IEwgNi4yODk5OTk5NjE4NTMwMjcgMTIuNDYwMDAwMDM4MTQ2OTczIEwgNi45NzAwMDAyNjcwMjg4MDkgMTEuNzcwMDAwNDU3NzYzNjcyIFoiIHN0eWxlPSJmaWxsOiByZ2IoMjU1LCAyNTUsIDI1NSk7Ii8%2BPHBhdGggZD0iTSA5LjAyOTk5OTczMjk3MTE5MSAxMS43NzAwMDA0NTc3NjM2NzIgTCA5LjcxMDAwMDAzODE0Njk3MyAxMi40NjAwMDAwMzgxNDY5NzMgTCAxNC4xNzAwMDAwNzYyOTM5NDUgOCBMIDkuNzEwMDAwMDM4MTQ2OTczIDMuNTM5OTk5OTYxODUzMDI3MyBMIDkuMDI5OTk5NzMyOTcxMTkxIDQuMjMwMDAwMDE5MDczNDg2IEwgMTIuODAwMDAwMTkwNzM0ODYzIDggTCA5LjAyOTk5OTczMjk3MTE5MSAxMS43NzAwMDA0NTc3NjM2NzIgWiIgc3R5bGU9ImZpbGw6IHJnYigyNTUsIDI1NSwgMjU1KTsiLz48cGF0aCBkPSJNOCwwYTgsOCwwLDEsMCw4LDhBOCw4LDAsMCwwLDgsMFpNOCwxNWE3LDcsMCwxLDEsNy03QTcsNywwLDAsMSw4LDE1WiIgc3R5bGU9ImZpbGw6IHJnYigyNTUsIDI1NSwgMjU1KTsiLz48L2c%2BPC9nPjwvZz48L3N2Zz4%3D)](https://github.com/Microsoft/BotBuilder) ``` ================================================ FILE: Contributing.md ================================================ # Instructions for Contributing Code ## Contributing bug fixes and features The Bot Framework team is currently accepting contributions in the form of bug fixes and new features. Any submission must have an issue tracking it in the issue tracker that has been approved by the Bot Framework team. Your pull request should include a link to the bug that you are fixing. If you've submitted a PR for a bug, please post a comment in the bug to avoid duplication of effort. ## Legal If your contribution is more than 15 lines of code, you will need to complete a Contributor License Agreement (CLA). Briefly, this agreement testifies that you are granting us permission to use the submitted change according to the terms of the project's license, and that the work being submitted is under appropriate copyright. Please submit a Contributor License Agreement (CLA) before submitting a pull request. You may visit https://cla.azure.com to sign digitally. Alternatively, download the agreement ([Microsoft Contribution License Agreement.docx](https://www.codeplex.com/Download?ProjectName=typescript&DownloadId=822190) or [Microsoft Contribution License Agreement.pdf](https://www.codeplex.com/Download?ProjectName=typescript&DownloadId=921298)), sign, scan, and email it back to . Be sure to include your github user name along with the agreement. Once we have received the signed CLA, we'll review the request. ================================================ FILE: LICENSE ================================================ The MIT License (MIT) Copyright (c) 2016 Microsoft Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. ================================================ FILE: Orchestrator/README.md ================================================ # Orchestrator Conversational AI applications today are built using disparate technologies to fulfill language understanding (LU) needs, such as [LUIS][1] and [QnA Maker][2]. Often, conversational AI applications are also built by accessing subsequent [skills][3], each of which handle a specific conversation topic and can be built using different LU technologies. Hence, conversational AI applications typically require LU to route an incoming user request to an appropriate skill or to dispatch to a specific sub-component. Orchestrator is an LU solution optimized for conversational AI applications. It is built ground-up to run locally with your bot. See the [technical overview][18] for additional details. ## Scenarios **Routing**: For bots, Orchestrator can replace the [LUIS Dispatch tool][5]. You can use Orchestrator instead of Dispatch to arbitrate between multiple [LUIS][1] and [QnA Maker][2] applications. With Orchestrator, you are likely to see: - Improved classification accuracy. - Higher resilience to data imbalance across your LUIS and QnA Maker authoring data. - Ability to correctly dispatch from relatively little authoring data. **Intent recognition**: You can use Orchestrator as an intent recognizer with [adaptive dialogs][6] to route user input to an appropriate skill or sub-component. **Entity extraction** is currently experimental and not yet for production use. ## Authoring experience Orchestrator can be used in different development environments: - [Bot Framework SDK][24]: Orchestrator can be integrated into your code project by replacing LUIS for intent recognition, such as for skill delegation or dispatching to subsequent language understanding services. See the [SDK integration](#sdk-integration) section for more information. - [Bot Framework Composer][19]: Orchestrator can be selected as a recognizer within Bot Framework Composer. At this point there are limitations to using Orchestrator in Composer, primarily around importing of existing models and tuning recognition performance. (To use Orchestrator, enable the feature flag in your Composer settings.) See the [Composer integration](#composer-integration) section for more information. In most cases, the [Bot Framework CLI][7] is required to prepare and optimize the model for your domain. The [BF Orchestrator command usage][23] page describes how to create, evaluate, and use an Orchestrator model. This diagram illustrates the first part of that process.

**Note**: To use the CLI, first install the [Bot Framework CLI][7]. See the [BF Orchestrator command usage][23] page for instructions on how to create and optimize the language model for your bot. ## SDK integration To use Orchestrator in place of Dispatch in an existing bot: - Create an _Orchestrator recognizer_ and provide it the path to the base model and your snapshot. - Use the recognizer's _recognize_ method to recognize user input. ### In a C\# bot - Install the `Microsoft.Bot.Builder.AI.Orchestrator` NuGet package. - Install the latest supported version of the [Visual C++ redistributable package](https://support.microsoft.com/help/2977003/the-latest-supported-visual-c-downloads). ```csharp using Microsoft.Bot.Builder.AI.Orchestrator; // Get Model and Snapshot path. string modelPath = Path.GetFullPath(OrchestratorConfig.ModelPath); string snapshotPath = Path.GetFullPath(OrchestratorConfig.SnapshotPath); // Create OrchestratorRecognizer. OrchestratorRecognizer orc = new OrchestratorRecognizer() { ModelPath = modelPath, SnapshotPath = snapshotPath }; // Recognize user input. var recoResult = await orc.RecognizeAsync(turnContext, cancellationToken); ``` ### In a JavaScript bot - Install the `botbuilder-ai-orchestrator` npm package to your bot. ```javascript const { OrchestratorRecognizer } = require('botbuilder-ai-orchestrator'); // Create OrchestratorRecognizer. const dispatchRecognizer = new OrchestratorRecognizer().configure({ modelPath: process.env.ModelPath, snapshotPath: process.env.SnapShotPath }); // To recognize user input const recoResult = await dispatchRecognizer.recognize(context); ``` ## Composer integration Orchestrator can be used as recognizer in [Bot Framework Composer][19]. In general, to specify Orchestrator as a dialog recognizer: 1. Select **Orchestrator** in the **Recognizer Type** drop-down menu for your bot. 2. Review, evaluate and adjust examples in language data as you would normally for LUIS to ensure recognition quality. This enables basic intent recognition. For more advanced scenarios follow the steps above to import and tune up routing quality. For more information about recognizers in Composer, see the discussion of [recognizers](https://docs.microsoft.com/composer/concept-dialog#recognizer) with respect to dialogs in Composer. Please make sure to follow the latest up-to-date instructions in [Composer documentation][25] ### Model Versions Composer is using the pre-selected default base models. With the CLI one can download & use alternate models. To force Composer to use different models, one can specify in the bot advanced settings as follows. The user can change the model by adding the following to appsettings.json: ``` "orchestrator": { "model": { "en_intent": "pretrained.20200924.microsoft.dte.00.03.en.onnx", "multilingual_intent": "pretrained.20200924.microsoft.dte.00.03.multilingual.onnx" } } ``` If this section is blank or unreadable, we simply use the Orchestrator defaults. If the model is not in the supported Orchestrator list, we throw an error message. See more on models [here][20]. ## Limitations * Orchestrator is limited to intents only. Entity definitions are ignored and no entity extraction is performed during recognition. ## Platform Support Orchestrator supports the following platforms. ### OS Support MacOS v10.14 / v10.15 Ubuntu 18 / 20 Windows 10 ### Language Support Nodejs v10, v12, v14 C# .NET Standard 2.1 C# .NET Core 3.1 ## Additional Reading - [Tech overview][18] - [API reference][14] - [Roadmap](./docs/Overview.md#Roadmap) - [BF CLI Orchestrator commands][11] - [C# samples][12] - [NodeJS samples][13] - [BF Orchestrator Command Usage][23] See the [School skill navigator](https://github.com/microsoft/BotBuilder-Samples/tree/main/composer-samples/csharp_dotnetcore/projects/OrchestratorSchoolNavigator) for an example of using Orchestrator commandlets to improve the quality of a .lu training set and using Composer to build a bot from examples in .lu format. [1]:https://luis.ai [2]:https://qnamaker.ai [3]:https://docs.microsoft.com/en-us/azure/bot-service/skills-conceptual [4]:https://en.wikipedia.org/wiki/Transformer_(machine_learning_model) [5]:https://docs.microsoft.com/azure/bot-service/bot-builder-tutorial-dispatch?tabs=cs [6]:https://aka.ms/adaptive-dialogs [7]:https://github.com/microsoft/botframework-cli [8]:https://github.com/microsoft/botframework-cli/tree/main/packages/luis#bf-luisversionexport [9]:https://github.com/microsoft/botframework-cli/tree/main/packages/luis#bf-luisconvert [10]:https://github.com/microsoft/botframework-cli/tree/main/packages/qnamaker#bf-qnamakerkbexport [11]:https://github.com/microsoft/botframework-cli/tree/main/packages/orchestrator [12]:https://github.com/microsoft/BotBuilder-Samples/tree/main/samples/csharp_dotnetcore/14.nlp-with-orchestrator [13]:https://github.com/microsoft/BotBuilder-Samples/tree/main/samples/javascript_nodejs/14.nlp-with-orchestrator [14]:https://aka.ms/bforchestratorapi [16]:https://github.com/microsoft/botframework-cli/tree/main/packages/orchestrator#bf-orchestratorcreate [18]:./docs/Overview.md [19]: https://docs.microsoft.com/composer/introduction [20]: https://aka.ms/NLRModels "Natural Language Representation Models" [21]:https://docs.microsoft.com/azure/bot-service/file-format/bot-builder-lu-file-format "LU file format" [22]:./docs/BFOrchestratorReport.md "report interpretation" [23]: ./docs/BFOrchestratorUsage.md "BF Orchestrator command usage" [24]:https://docs.microsoft.com/azure/bot-service/index-bf-sdk [25]: https://github.com/microsoft/BotFramework-Composer/blob/main/docs/preview%20features/orchestrator.md "Composer integration instructions" ================================================ FILE: Orchestrator/Samples/CLI/ModelTuning/README.md ================================================ # Improve Language Model with BF CLI The following sample illustrates how to evaluate and improve a simple language model using the Orchestrator report. You may adopt this process simply by editing the [demo.cmd](./demo.cmd) script to use your language model files. ## Prerequisites * [Bot Framework CLI][5] * [Bot Framework CLI Orchestrator plugin][1] * An understanding of [Orchestrator][6] feature usage. ## Walkthrough The following files make up this illustration: ``` demo.cmd: Script run evaluation and produces report common.lu: Bot Language Model LU common.test.lu: A test set containing examples not present in original LU common.fixed.lu: A corrected Language Model based on evaluation run ``` Assume a bot with a simple language model in [common.lu](./common.lu) file as follows: ``` # Help - help - i need help - please help - can you please help - what do you do - what is this bot for # BuySurface - How can I buy {ProductType=Surface PRO} - I want to buy {ProductType=Surface PRO} - I want to buy {ProductType=Surface laptop} - Can I buy {ProductType=Surface PRO} online # USConstitution - Tell me about the US Constitution - US Constitution - Info on the Constitution - tell me about the Constitution - constitution knowledgebase - constitution faq - what do you know about the constitution - The united states constitution # Restart - restart - reset - start over - menu - top menu # Finance - pay my bills - transfer funds - what's my balance - get $500 cash # LUIS - luis # Orchestrator - Orchestrator - Orch ``` This evaluation examines how the Orchestrator engine can generalize language understanding from a few examples to include phrases never seen before. Hence, for proper language evaluation, the test file [common.test.lu](./common.test.lu) should contain utterances not present in the original "training" set: ``` # Help - help - do you have help - any assistance in this bot? # BuySurface - Looking for a computer made by MSFT - what kind of MS products do you have? # USConstitution - Where can I read our founding fathers famous document? - Is there an analysis of the bill of rights? - What is that proclamation thing? # Restart - reset - go to the beginning # Finance - can i pay with credit card? - do you use Zelle? - what is my bank borrowing limit? # LUIS - Language Understanding Inteligent Service # Orchestrator - Conductor ``` Next, run the [demo.cmd](./demo.cmd) script. In first run you need to download the Orchestrator basemodel so execute ```demo.cmd getmodel``` which will download the model and run the test. Consequent runs can reuse the same basemodel (hint: see ```bf orchestrator:basemodel:list``` for alternate models). Also, if you wish to compare to [LUIS](https://luis.ai) results, edit the script with your LUIS application info. This will produce a report in report folder such as follows: ![Before Correction](./reportbefore.png) Notice how 5 utterances were misclassified. For example ```do you use Zelle?``` was classified as *Help* (with low score of 0.2998) instead of *Finance*. Also, note that what led to this misclassification is that the nearest example scored 0.5669 under the *Help* label. The observation shows that the concepts for those misclassified utterances are not present in the original common.lu language model. To correct, we'll add representative examples in the original common.lu file. Note how we don't add the exact utterances, only representative examples: ![Compared LU](./comparedLU.png) The corrected file is available for reference as [common.fixed.lu](./common.fixed.lu). You may copy it over the common.lu. Now, re-run the test ```demo.cmd``` and view the resulting report: ![Report After](./reportafter.png) ## Summary This walkthrough showed how to improve your bot's language model before ever deploying it. It uses BF CLI to test the language model, and use the resulting report to correct the language model. One should construct a language model with as many examples representing expected user utterances. However, it also illustrates how one does not need to account for all utterance permutations, rather only to present concepts within those utterances. The Orchestrator engine generalizes and can accurately detect similar utterances. ## Additional Reading - [Orchestrator Documentation][6] - [BF CLI Orchestrator Command Usage][4] - [Report Interpretation][3] - [LU File Format][2] [1]: https://aka.ms/bforchestratorcli "Orchestrator Plugin" [2]: https://docs.microsoft.com/en-us/azure/bot-service/file-format/bot-builder-lu-file-format?view=azure-bot-service-4.0 "LU file format" [3]: https://aka.ms/bforchestratorreport "report interpretation" [4]: https://github.com/microsoft/botframework-sdk/blob/main/Orchestrator/docs/BFOrchestratorUsage.md "BF Orchestrator usage" [5]: https://github.com/microsoft/botframework-cli [6]: https://aka.ms/bf-orchestrator ================================================ FILE: Orchestrator/Samples/CLI/ModelTuning/common.fixed.lu ================================================ # Help - help - i need help - please help - can you please help - what do you do - what is this bot for # BuySurface - How can I buy {ProductType=Surface PRO} - I want to buy {ProductType=Surface PRO} - I want to buy {ProductType=Surface laptop} - Can I buy {ProductType=Surface PRO} online - Types of MS products? # USConstitution - Tell me about the US Constitution - US Constitution - Info on the Constitution - tell me about the Constitution - constitution knowledgebase - constitution faq - what do you know about the constitution - The united states constitution # Restart - restart - reset - start over - menu - top menu # Finance - pay my bills - transfer funds - what's my balance - get $500 cash - use Zelle - borrowing limits? # LUIS - luis - Language Understanding # Orchestrator - Orchestrator - Orch - Conductor organizer ================================================ FILE: Orchestrator/Samples/CLI/ModelTuning/common.lu ================================================ # Help - help - i need help - please help - can you please help - what do you do - what is this bot for # BuySurface - How can I buy {ProductType=Surface PRO} - I want to buy {ProductType=Surface PRO} - I want to buy {ProductType=Surface laptop} - Can I buy {ProductType=Surface PRO} online # USConstitution - Tell me about the US Constitution - US Constitution - Info on the Constitution - tell me about the Constitution - constitution knowledgebase - constitution faq - what do you know about the constitution - The united states constitution # Restart - restart - reset - start over - menu - top menu # Finance - pay my bills - transfer funds - what's my balance - get $500 cash # LUIS - luis # Orchestrator - Orchestrator - Orch ================================================ FILE: Orchestrator/Samples/CLI/ModelTuning/common.test.lu ================================================ # Help - help - do you have help - any assistance in this bot? # BuySurface - Looking for a computer made by MSFT - what kind of MS products do you have? # USConstitution - Where can I read our founding fathers famous document? - Is there an analysis of the bill of rights? - What is that proclamation thing? # Restart - reset - go to the beginning # Finance - can i pay with credit card? - do you use Zelle? - what is my bank borrowing limit? # LUIS - Language Understanding Inteligent Service # Orchestrator - Conductor ================================================ FILE: Orchestrator/Samples/CLI/ModelTuning/demo.cmd ================================================ @echo off @echo ORCHESTRATOR EVALUATION DEMO @rem set SEED for different test sets set SEED=common set BLU=generated\%SEED%.blu set LUFILE=%SEED%.lu @rem test file contains sample utterances that are not in main LU file. set TESTFILE=%SEED%.test.lu @rem proper test @rem set TESTFILE=%SEED%.test.lu set LUISKEY= set LUISAPP= set LUISHOST= set SKIPLUIS=0 set QUERYRUN=0 if "%LUISKEY%" == "" ( @echo Skipping comparison with LUIS. Fill in LUIS info to compare results. set SKIPLUIS=1 ) @rem set QUERY to run a single utterance test set QUERY="what is the american declaration of independence?" if "%1" == "qonly" ( set QUERYRUN=1 goto QUERYONLY ) @rem model folder needs to be downloaded only once. if "%1" == "getmodel" ( if EXIST model rd /s /q model ) @echo cleaning folders if EXIST report ( rd /s /q report && md report ) if EXIST generated ( rd /s /q generated && md generated ) @rem Only need to retrieve model once IF NOT EXIST .\model ( @rem see bf orchestrator:basemodel:get --help to get the non-default model @rem see available models via bf orchestrator:basemodel:list @echo getting base model md model call bf orchestrator:basemodel:get --out model ) @echo Create orchestrator snapshot .blu file call bf orchestrator:create --model model --out %BLU% --in %LUFILE% @echo running orchestrator test to generate a report (see report folder) call bf orchestrator:test --in %BLU% --model ./model --out report --test %TESTFILE% if "%SKIPLUIS%" == "0" ( @echo running LUIS test... call bf luis:test --subscriptionKey %LUISKEY% --endpoint %LUISHOST% --appId %LUISAPP% --in %TESTFILE% --out report/luisresult.txt ) :QUERYONLY @rem Illustrates how to query for only a single utterance. Edit %QUERY% above. if "%QUERYRUN%" == "1" ( echo Orchestrator single utterance query: echo bf orchestrator:query --in %BLU% --model model --query %QUERY% call bf orchestrator:query --in %BLU% --model model --query %QUERY% if "%SKIPLUIS%" == "0" ( echo LUIS single utterance query: echo bf luis:application:query --appId LUISAPP --endpoint LUISHOST --subscriptionKey LUISKEY --query %QUERY% call bf luis:application:query --appId %LUISAPP% --endpoint %LUISHOST% --subscriptionKey %LUISKEY% --query %QUERY% ) ) :DONE ================================================ FILE: Orchestrator/Samples/dotnet/nlp-with-entities/.gitignore ================================================ # Assets generated by luis:build, luis:cross-train **/model **/[Gg]enerated **/*.blu *.json **/*.targets **/*.props **/*.cache *.dgspec.json ================================================ FILE: Orchestrator/Samples/dotnet/nlp-with-entities/AdapterWithErrorHandler.cs ================================================ // Copyright (c) Microsoft Corporation. All rights reserved. // Licensed under the MIT License. using System; using Microsoft.Bot.Builder; using Microsoft.Bot.Builder.Integration.AspNet.Core; using Microsoft.Bot.Builder.TraceExtensions; using Microsoft.Extensions.Configuration; using Microsoft.Extensions.Logging; namespace Microsoft.BotBuilderSamples { public class AdapterWithErrorHandler : BotFrameworkHttpAdapter { public AdapterWithErrorHandler(IConfiguration configuration, ILogger logger, ConversationState conversationState = null) : base(configuration, logger) { OnTurnError = async (turnContext, exception) => { // Log any leaked exception from the application. logger.LogError(exception, $"[OnTurnError] unhandled error : {exception.Message}"); // Send a message to the user await turnContext.SendActivityAsync("The bot encountered an error or bug."); await turnContext.SendActivityAsync("To run this sample make sure you have the LUIS and QnA models deployed."); await turnContext.SendActivityAsync("To continue to run this bot, please fix the bot source code."); if (conversationState != null) { try { // Delete the conversationState for the current conversation to prevent the // bot from getting stuck in a error-loop caused by being in a bad state. // ConversationState should be thought of as similar to "cookie-state" in a Web pages. await conversationState.DeleteAsync(turnContext); } catch (Exception e) { logger.LogError(e, $"Exception caught on attempting to Delete ConversationState : {e.Message}"); } } // Send a trace activity, which will be displayed in the Bot Framework Emulator await turnContext.TraceActivityAsync("OnTurnError Trace", exception.Message, "https://www.botframework.com/schemas/error", "TurnError"); }; } } } ================================================ FILE: Orchestrator/Samples/dotnet/nlp-with-entities/BotServices.cs ================================================ // Copyright (c) Microsoft Corporation. All rights reserved. // Licensed under the MIT License. using Microsoft.Bot.Builder.AI.Luis; using Microsoft.Bot.Builder.AI.Orchestrator; using Microsoft.Bot.Builder.AI.QnA; using Microsoft.Extensions.Configuration; namespace Microsoft.BotBuilderSamples { public class BotServices : IBotServices { public BotServices(IConfiguration configuration, OrchestratorRecognizer dispatcher) { // Read the setting for cognitive services (LUIS, QnA) from the appsettings.json // If includeApiResults is set to true, the full response from the LUIS api (LuisResult) // will be made available in the properties collection of the RecognizerResult LuisHomeAutomationRecognizer = CreateLuisRecognizer(configuration, "LuisHomeAutomationAppId"); LuisWeatherRecognizer = CreateLuisRecognizer(configuration, "LuisWeatherAppId"); Dispatch = dispatcher; SampleQnA = new QnAMaker(new QnAMakerEndpoint { KnowledgeBaseId = configuration["QnAKnowledgebaseId"], EndpointKey = configuration["QnAEndpointKey"], Host = configuration["QnAEndpointHostName"] }); } public OrchestratorRecognizer Dispatch { get; private set; } public QnAMaker SampleQnA { get; private set; } public LuisRecognizer LuisHomeAutomationRecognizer { get; private set; } public LuisRecognizer LuisWeatherRecognizer { get; private set; } private LuisRecognizer CreateLuisRecognizer(IConfiguration configuration, string appIdKey) { var luisApplication = new LuisApplication( configuration[appIdKey], configuration["LuisAPIKey"], configuration["LuisAPIHostName"]); // Set the recognizer options depending on which endpoint version you want to use. // More details can be found in https://docs.microsoft.com/en-gb/azure/cognitive-services/luis/luis-migration-api-v3 var recognizerOptions = new LuisRecognizerOptionsV2(luisApplication) { IncludeAPIResults = true, PredictionOptions = new LuisPredictionOptions() { IncludeAllIntents = true, IncludeInstanceData = true } }; return new LuisRecognizer(recognizerOptions); } } } ================================================ FILE: Orchestrator/Samples/dotnet/nlp-with-entities/Bots/DispatchBot.cs ================================================ // Copyright (c) Microsoft Corporation. All rights reserved. // Licensed under the MIT License. using System.Collections.Generic; using System.Linq; using System.Threading; using System.Threading.Tasks; using Microsoft.Azure.CognitiveServices.Language.LUIS.Runtime.Models; using Microsoft.Bot.Builder; using Microsoft.Bot.Builder.Dialogs; using Microsoft.Bot.Schema; using Microsoft.Extensions.Logging; namespace Microsoft.BotBuilderSamples { public class DispatchBot : ActivityHandler { private ILogger _logger; private IBotServices _botServices; public DispatchBot(IBotServices botServices, ILogger logger) { _logger = logger; _botServices = botServices; } protected override async Task OnMessageActivityAsync(ITurnContext turnContext, CancellationToken cancellationToken) { var dc = new DialogContext(new DialogSet(), turnContext, new DialogState()); // Top intent tell us which cognitive service to use. var allScores = await _botServices.Dispatch.RecognizeAsync(dc, (Activity)turnContext.Activity, cancellationToken); var topIntent = allScores.Intents.First().Key; // Detected entities which could be used to help with intent dispatching, for example, when top intent score is too low or // when there are multiple top intents with close scores. var entities = allScores.Entities.Children().Where(t => t.Path != "$instance"); // Next, we call the dispatcher with the top intent. await DispatchToTopIntentAsync(turnContext, topIntent, cancellationToken); } protected override async Task OnMembersAddedAsync(IList membersAdded, ITurnContext turnContext, CancellationToken cancellationToken) { const string WelcomeText = "Type a greeting, or a question about the weather to get started."; foreach (var member in membersAdded) { if (member.Id != turnContext.Activity.Recipient.Id) { await turnContext.SendActivityAsync(MessageFactory.Text($"Welcome to Dispatch bot {member.Name}. {WelcomeText}"), cancellationToken); } } } private async Task DispatchToTopIntentAsync(ITurnContext turnContext, string intent, CancellationToken cancellationToken) { switch (intent) { case "HomeAutomation": await ProcessHomeAutomationAsync(turnContext, cancellationToken); break; case "Weather": await ProcessWeatherAsync(turnContext, cancellationToken); break; case "QnAMaker": await ProcessSampleQnAAsync(turnContext, cancellationToken); break; default: _logger.LogInformation($"Dispatch unrecognized intent: {intent}."); await turnContext.SendActivityAsync(MessageFactory.Text($"Dispatch unrecognized intent: {intent}."), cancellationToken); break; } } private async Task ProcessHomeAutomationAsync(ITurnContext turnContext, CancellationToken cancellationToken) { _logger.LogInformation("ProcessHomeAutomationAsync"); // Retrieve LUIS result for HomeAutomation. var recognizerResult = await _botServices.LuisHomeAutomationRecognizer.RecognizeAsync(turnContext, cancellationToken); var result = recognizerResult.Properties["luisResult"] as LuisResult; var topIntent = result.TopScoringIntent.Intent; await turnContext.SendActivityAsync(MessageFactory.Text($"HomeAutomation top intent {topIntent}."), cancellationToken); await turnContext.SendActivityAsync(MessageFactory.Text($"HomeAutomation intents detected:\n\n{string.Join("\n\n", result.Intents.Select(i => i.Intent))}"), cancellationToken); if (result.Entities.Count > 0) { await turnContext.SendActivityAsync(MessageFactory.Text($"HomeAutomation entities were found in the message:\n\n{string.Join("\n\n", result.Entities.Select(i => i.Entity))}"), cancellationToken); } } private async Task ProcessWeatherAsync(ITurnContext turnContext, CancellationToken cancellationToken) { _logger.LogInformation("ProcessWeatherAsync"); // Retrieve LUIS result for Weather. var recognizerResult = await _botServices.LuisWeatherRecognizer.RecognizeAsync(turnContext, cancellationToken); var result = recognizerResult.Properties["luisResult"] as LuisResult; var topIntent = result.TopScoringIntent.Intent; await turnContext.SendActivityAsync(MessageFactory.Text($"ProcessWeather top intent {topIntent}."), cancellationToken); await turnContext.SendActivityAsync(MessageFactory.Text($"ProcessWeather Intents detected::\n\n{string.Join("\n\n", result.Intents.Select(i => i.Intent))}"), cancellationToken); if (result.Entities.Count > 0) { await turnContext.SendActivityAsync(MessageFactory.Text($"ProcessWeather entities were found in the message:\n\n{string.Join("\n\n", result.Entities.Select(i => i.Entity))}"), cancellationToken); } } private async Task ProcessSampleQnAAsync(ITurnContext turnContext, CancellationToken cancellationToken) { _logger.LogInformation("ProcessSampleQnAAsync"); var results = await _botServices.SampleQnA.GetAnswersAsync(turnContext); if (results.Any()) { await turnContext.SendActivityAsync(MessageFactory.Text(results.First().Answer), cancellationToken); } else { await turnContext.SendActivityAsync(MessageFactory.Text("Sorry, could not find an answer in the Q and A system."), cancellationToken); } } } } ================================================ FILE: Orchestrator/Samples/dotnet/nlp-with-entities/CognitiveModels/HomeAutomation.lu ================================================ > LUIS application information > !# @app.name = Home Automation > !# @app.desc = Home Automation LUIS application - Bot Builder Samples > !# @app.versionId = 0.1 > !# @app.culture = en-us > !# @app.luis_schema_version = 4.0.0 > !# @app.tokenizerVersion = 1.0.0 > # Intent definitions ## ControlDevice - {@Room=breezeway} on please - change {@deviceProperty=temperature} to seventy two degrees - {@Device=coffee bar} on please - decrease {@deviceProperty=temperature} for me please - dim {@Room=kitchen} {@Device=lights} to 25 . - {@Device=fish pond} off please - {@Device=fish pond} on please - {@Device=illuminate} please - {@Room=living room} {@Device=lamp} on please - {@Room=living room} {@Device=lamps} off please - lock the {@Device=doors} for me please - lower your {@deviceProperty=volume} - make {@Device=camera 1} off please - make some {@Device=coffee} - play {@Device=dvd} - set {@Device=lights} out in {@Room=bedroom} - set {@Device=lights} to {@deviceProperty=bright} - set {@Device=lights} to {@deviceProperty=concentrate} - shut down my work {@Device=computer} - snap switch {@Device=fan} fifty percent - start {@Room=master bedroom} {@Device=light}. - {@Room=theater} on please - turn {@Device=dimmer} off - turn off {@Device=ac} please - turn off {@Room=foyer} {@Device=lights} - turn off {@Room=living room} {@Device=light} - turn off {@Device=staircase} - turn off venice {@Device=lamp} - turn on {@Room=bathroom} {@Device=heater} - turn on {@Device=external speaker} - turn on {@Room=kitchen} {@Device=faucet} - turn on {@Device=light} in {@Room=bedroom} - turn on my {@Room=bedroom} {@Device=lights}. - turn on the {@Room=furnace room} {@Device=lights} - turn on the internet in my {@Room=bedroom} please - turn on {@Device=thermostat} please - turn the {@Device=fan} to high - turn {@Device=thermostat} on 70. - {@Device_PatternAny} off [please] - {@Device_PatternAny} in {Room_PatternAny} on [please] - turn on {@Device_PatternAny} in {Room_PatternAny} - turn off {@Device_PatternAny} in {Room_PatternAny} - {@Device_PatternAny} in {Room_PatternAny} off [please] - {@Device_PatternAny} on [please] - turn on {@Device_PatternAny} ## None > # Entity definitions @ ml Device @ ml deviceProperty @ ml Room > # PREBUILT Entity definitions @ prebuilt number > # Phrase list definitions > # List entities @ list Operation = - off : - turn off - switch off - lock - out - shut down - stop - on : - turn on - switch on - unlock - un lock - boot up - start > # RegEx entities > # Pattern.Any entities @ patternany Device_PatternAny @ patternany Room_PatternAny ================================================ FILE: Orchestrator/Samples/dotnet/nlp-with-entities/CognitiveModels/QnAMaker.qna ================================================ # ? hi ``` hello ``` # ? greetings - good morning - good evening ``` Hello! ``` # ? What are you? - What? - What do you do? - Who are you? - What is your name? - What should I call you? ``` I am the LUIS-QnAMaker Dispatch bot! This sample demonstrates using several LUIS applications and QnA Maker knowledge base using Orchestrator. ``` ================================================ FILE: Orchestrator/Samples/dotnet/nlp-with-entities/CognitiveModels/Weather.lu ================================================ > LUIS application information > !# @app.name = Weather > !# @app.desc = Weather LUIS application - Bot Builder Samples > !# @app.versionId = 0.1 > !# @app.culture = en-us > !# @app.luis_schema_version = 4.0.0 > # Intent definitions ## Get Weather Condition - current weather ? - do {@Location=florida} residents usually need ice scrapers - get {@Location=florence} temperature in september - get for me the weather conditions in {@Location=sonoma county} - get the daily temperature {@Location=greenwood indiana} - get the weather at {@Location=saint george utah} - how much rain does {@Location=chambersburg} get a year - show average rainfall for {@Location=boise} - temperature of {@Location=delhi} in celsius please - was last year about this time as wet as it is now in the {@Location=south} ? - what is the rain volume in {@Location=sonoma county} ? - what to wear in march in {@Location=california} - what's the weather like in {@Location=minneapolis} - weather in {@Location_PatternAny} - how's the weather in {@Location_PatternAny} - current weather in {@Location_PatternAny} ## Get Weather Forecast - forecast in celcius - get the forcast for me - i want to know the temperature at {@Location=death valley} - provide me by {@Location=toronto} weather please - show me the forecast at {@Location=alabama} - soliciting today's weather - what is the weather in {@Location=redmond} ? - what is the weather today at 10 day {@Location=durham} ? - what will the weather be tomorrow in {@Location=new york} ? - what's the weather going to be like in {@Location=hawaii} ? - will it be raining in {@Location=ranchi} - will it rain this weekend - will it snow today - what's the forecast for next week in {@Location_PatternAny} - show me the forecast for {@Location_PatternAny} - what's the forecast for {@Location_PatternAny} ## None > # Entity definitions @ ml Location > # PREBUILT Entity definitions > # Phrase list definitions > # List entities > # RegEx entities > # Pattern.Any entities @ patternany Location_PatternAny ================================================ FILE: Orchestrator/Samples/dotnet/nlp-with-entities/Controllers/BotController.cs ================================================ // Copyright (c) Microsoft Corporation. All rights reserved. // Licensed under the MIT License. using System.Threading.Tasks; using Microsoft.AspNetCore.Mvc; using Microsoft.Bot.Builder; using Microsoft.Bot.Builder.Integration.AspNet.Core; namespace Microsoft.BotBuilderSamples { [Route("api/messages")] [ApiController] public class BotController : ControllerBase { private IBotFrameworkHttpAdapter _adapter; private IBot _bot; public BotController(IBotFrameworkHttpAdapter adapter, IBot bot) { _adapter = adapter; _bot = bot; } [HttpPost] public async Task PostAsync() { // Delegate the processingg of the HTTP POST to the adapter. // The adapter will invoke the bot. await _adapter.ProcessAsync(Request, Response, _bot); } } } ================================================ FILE: Orchestrator/Samples/dotnet/nlp-with-entities/IBotServices.cs ================================================ // Copyright (c) Microsoft Corporation. All rights reserved. // Licensed under the MIT License. using Microsoft.Bot.Builder.AI.Luis; using Microsoft.Bot.Builder.AI.Orchestrator; using Microsoft.Bot.Builder.AI.QnA; namespace Microsoft.BotBuilderSamples { public interface IBotServices { LuisRecognizer LuisHomeAutomationRecognizer { get; } LuisRecognizer LuisWeatherRecognizer { get; } OrchestratorRecognizer Dispatch { get; } QnAMaker SampleQnA { get; } } } ================================================ FILE: Orchestrator/Samples/dotnet/nlp-with-entities/OrchestratorConfig.cs ================================================ namespace Microsoft.BotBuilderSamples { public class OrchestratorConfig { public string SnapshotFile { get; set; } public string ModelFolder { get; set; } } } ================================================ FILE: Orchestrator/Samples/dotnet/nlp-with-entities/Program.cs ================================================ // Copyright (c) Microsoft Corporation. All rights reserved. // Licensed under the MIT License. using Microsoft.AspNetCore; using Microsoft.AspNetCore.Hosting; using Microsoft.Extensions.Logging; namespace Microsoft.BotBuilderSamples { public class Program { public static void Main(string[] args) { CreateWebHostBuilder(args).Build().Run(); } public static IWebHostBuilder CreateWebHostBuilder(string[] args) => WebHost.CreateDefaultBuilder(args) .ConfigureLogging((logging) => { logging.AddDebug(); logging.AddConsole(); }) .UseStartup(); } } ================================================ FILE: Orchestrator/Samples/dotnet/nlp-with-entities/README.md ================================================ # NLP with Orchestrator Bot Framework v4 NLP with Orchestrator (with entity recognition) bot sample This bot has been created using [Bot Framework](https://dev.botframework.com), it shows how to create a bot that relies on multiple [LUIS.ai](https://www.luis.ai) and [QnAMaker.ai](https://www.qnamaker.ai) models for natural language processing (NLP). Use the Orchestrator dispatch model in cases when: - Your bot consists of multiple language modules (LUIS + QnA) and you need assistance in routing user's utterances to these modules in order to integrate the different modules into your bot. - Create a text classification model from text files. ## Overview This bot uses Orchestrator to route user utterances to multiple LUIS models and QnA maker services to support multiple conversational scenarios. This bot sample also demonstrates using Orchestrator for entity recognition. ## Prerequisites | OS | Version | Architectures | | ------- | ------------------- | --------------- | | Windows | 10 (1607+) | ia32 (x86), x64 | | MacOS | 10.15+ | x64 | | Linux | Ubuntu 18.04, 20.04 | x64 | This sample **requires** prerequisites in order to run. - Install latest supported version of [Visual C++ Redistributable](https://support.microsoft.com/en-gb/help/2977003/the-latest-supported-visual-c-downloads) - Install latest [Bot Framework Emulator](https://github.com/microsoft/BotFramework-Emulator/releases) - [.NET Core SDK](https://aka.ms/dotnet-core-applaunch?framework=Microsoft.AspNetCore.App&framework_version=3.1.0&arch=x64&rid=win10-x64) version 3.1 ```bash > dotnet --version ``` - Node v14 ```bash > node -v ``` - Install BF CLI with Orchestrator plugin ```bash > npm i -g @microsoft/botframework-cli ``` Make sure bf orchestrator command is working and shows all available orchestrator commands ```bash > bf orchestrator ``` ## To try this bot sample - Clone the repository ```bash > git clone https://github.com/microsoft/botbuilder-samples.git ``` - CD Orchestrator\Samples\dotnet\nlp-with-entities ```bash > cd Orchestrator\Samples\dotnet\nlp-with-entities ``` - Configure the LUIS applications (HomeAutomation and Weather) required for this sample. - Get your [LUIS authoring key](https://docs.microsoft.com/en-us/azure/cognitive-services/LUIS/luis-concept-keys) ```bash > bf luis:build --in CognitiveModels --authoringKey --botName ``` - Update application settings in `./appsettings.json`. - Configure the QnA Maker KB required for this sample. - Get your [QnA Maker Subscription key](https://docs.microsoft.com/en-us/azure/cognitive-services/QnAMaker/how-to/set-up-qnamaker-service-azure#create-a-new-qna-maker-service) ```bash > bf qnamaker:build --in CognitiveModels --subscriptionKey --botName ``` - Update kb information in `./appsettings.json` - Configure Orchestrator to route utterances to LUIS/QnA language services set up above - Download Orchestrator intent and entity base models ```bash > mkdir model > bf orchestrator:basemodel:get --out ./model > bf orchestrator:basemodel:get --versionId pretrained.20210218.microsoft.dte.00.06.bert_example_ner.en.onnx --out ./model/entity ``` - Create the Orchestrator snapshot ```bash > mkdir generated > bf orchestrator:create --hierarchical --in ./CognitiveModels --out ./generated --model ./model --entityModel ./model/entity ``` The *hierarchical* flag creates top level intents in the snapshot file derived from the .lu/.qna file names in the input folder. As a result, the example utterances are mapped to *HomeAutomation*, *QnAMaker* and *Weather* intents/labels. The optional *entityModel* parameter provides path to a pretrained base model for named entity recognition and will be used to compute entity embeddings for Orchestrator snapshot file. - Verify appsettings.json has the following: ``` "Orchestrator": { "ModelFolder": ".\\model", "SnapshotFile": ".\\generated\\orchestrator.blu" } ``` - Run the bot from a terminal or from Visual Studio, choose option A or B. A) From a terminal ```bash > cd samples\csharp_dotnetcore\14.nlp-with-orchestrator > dotnet run ``` B) Or from Visual Studio - Launch Visual Studio - File -> Open -> Project/Solution - Navigate to `Orchestrator` folder - Select `OrchestratorSamples.sln` file - Right click on `01.dispatch-bot` project in the solution and 'Set as Startup Project' - Press `F5` to run the project ## Connect to the bot using Bot Framework Emulator - Launch Bot Framework Emulator - File -> Open Bot - Enter a Bot URL of `http://localhost:3978/api/messages` - Try entering several utterances related to home automation, weather or some chit chat. Some examples: - set lights to bright - good morning - how is the weather in seattle ## Further reading - [Dispatch Migration Example](https://github.com/microsoft/botframework-sdk/blob/main/Orchestrator/docs/DispatchMigrationExample.md) - [Bot Framework Documentation](https://docs.botframework.com) - [BF Orchestrator Command Usage](https://github.com/microsoft/botframework-sdk/blob/main/Orchestrator/docs/BFOrchestratorUsage.md) - [Bot Basics](https://docs.microsoft.com/azure/bot-service/bot-builder-basics?view=azure-bot-service-4.0) - [Activity processing](https://docs.microsoft.com/en-us/azure/bot-service/bot-builder-concept-activity-processing?view=azure-bot-service-4.0) - [Azure Bot Service Introduction](https://docs.microsoft.com/azure/bot-service/bot-service-overview-introduction?view=azure-bot-service-4.0) - [Azure Bot Service Documentation](https://docs.microsoft.com/azure/bot-service/?view=azure-bot-service-4.0) - [.NET Core CLI tools](https://docs.microsoft.com/en-us/dotnet/core/tools/?tabs=netcore2x) - [Azure CLI](https://docs.microsoft.com/cli/azure/?view=azure-cli-latest) - [Azure Portal](https://portal.azure.com) - [Channels and Bot Connector Service](https://docs.microsoft.com/en-us/azure/bot-service/bot-concepts?view=azure-bot-service-4.0) ================================================ FILE: Orchestrator/Samples/dotnet/nlp-with-entities/Startup.cs ================================================ // Copyright (c) Microsoft Corporation. All rights reserved. // Licensed under the MIT License. using System.IO; using Microsoft.AspNetCore.Builder; using Microsoft.AspNetCore.Hosting; using Microsoft.AspNetCore.Mvc; using Microsoft.Bot.Builder; using Microsoft.Bot.Builder.AI.Orchestrator; using Microsoft.Bot.Builder.Integration.AspNet.Core; using Microsoft.Extensions.Configuration; using Microsoft.Extensions.DependencyInjection; using Microsoft.Extensions.Hosting; namespace Microsoft.BotBuilderSamples { public class Startup { public Startup(IConfiguration configuration) { OrchestratorConfig = configuration.GetSection("Orchestrator").Get(); } public OrchestratorConfig OrchestratorConfig { get; } // This method gets called by the runtime. Use this method to add services to the container. public void ConfigureServices(IServiceCollection services) { services.AddMvc().SetCompatibilityVersion(CompatibilityVersion.Latest); // Create the Bot Framework Adapter with error handling enabled. services.AddSingleton(); services.AddSingleton(InitializeOrchestrator()); // Create the bot services (LUIS, QnA) as a singleton. services.AddSingleton(); // Create the bot as a transient. services.AddTransient(); } // This method gets called by the runtime. Use this method to configure the HTTP request pipeline. public void Configure(IApplicationBuilder app, IWebHostEnvironment env) { if (env.IsDevelopment()) { app.UseDeveloperExceptionPage(); } app.UseDefaultFiles() .UseStaticFiles() .UseWebSockets() .UseRouting() .UseAuthorization() .UseEndpoints(endpoints => { endpoints.MapControllers(); }); } private OrchestratorRecognizer InitializeOrchestrator() { string modelFolder = Path.GetFullPath(OrchestratorConfig.ModelFolder); string snapshotFile = Path.GetFullPath(OrchestratorConfig.SnapshotFile); OrchestratorRecognizer orc = new OrchestratorRecognizer() { ModelFolder = modelFolder, SnapshotFile = snapshotFile }; return orc; } } } ================================================ FILE: Orchestrator/Samples/dotnet/nlp-with-entities/nlp-with-entities.csproj ================================================  netcoreapp3.1 latest x64 PreserveNewest PreserveNewest PreserveNewest PreserveNewest PreserveNewest ================================================ FILE: Orchestrator/Samples/dotnet/nlp-with-entities/wwwroot/default.html ================================================  Dispatch bot sample with Orchestrator
NLP With Dispatch Sample
Your bot is ready!
You can test your bot in the Bot Framework Emulator
by connecting to http://localhost:3978/api/messages.
Download the Emulator
Visit Azure Bot Service to register your bot and add it to
various channels. The bot's endpoint URL typically looks like this:
https://your_bots_hostname/api/messages
================================================ FILE: Orchestrator/docs/BFOrchestratorInteractive.md ================================================ # Interactive [BF Orchestrator CLI][1] has an "interactive" command which enables a user to dynamically interact with an Orchestrator base language model (see examples in [Start an interactive session without a training set](#start-an-interactive-session-without-a-training-set)) and improve the accuracy of an existing language model (see examples in [Start an interactive session with a training set](#start-an-interactive-session-with-a-training-set)) through some CLI commandlets. During an interactive session, the command loads an Orchestrator base model into memory and manage several input cache variables that can be used by ensuing commandlets for maintaining the base model's example set. These variables include: - **"current" utterance** -- A cache for storing an utterance that can be used by several commandlets for interacting with an Orchestrator model. - **"current" intent labels** -- A cache for storing an array of intent labels that can be used by several commandlet for interacting with an Orchestrator model. - **"new" intent labels** -- Another cache for storing an array of intent labels, which were mainly used for changing an utterance's intent labels within an Orchestrator model. ## Scenarios ### Start an interactive session without a training set An Orchestrator user can launch the interactive command without a training set. During an session, the user can interactively add utterance examples, revise them, remove them, validate and create an evaluation report, etc.. #### Argument setup Below is a command snippet for a user to start the interactive command with two arguments: - "--model" -- folder pointing to an Orchestrator base model - "--out" -- out folder for validation evaluation reports. ``` > set ORCHESTRATOR_MODEL= > set EVALUATING_OUTPUT= > bf orchestrator:interactive --out=%EVALUATING_OUTPUT% --model=%ORCHESTRATOR_MODEL% ``` After executing the command snippet, the command enters an interactive session shown below: ``` > "Current" utterance: "" > "Current" intent label array: "" > "New" intent label array: "" Please enter a commandlet, "h" for help > ``` #### Cache an utterance The user can then use the 'u' commandlet to enter an utterance and cache it into the "current" utterance variable. Below is what happened after a user enters "hi" as a new utterance using the 'u' commandlet. ``` Please enter a commandlet, "h" for help > u Please enter an utterance > hi > "Current" utterance: "hi" > "Current" intent label array: "" > "New" intent label array: "" Please enter a commandlet, "h" for help > ``` #### Cache an intent label Then the user can issue the 'i' commandlet to enter a new intent label and cache it into the "current" intent label variable. Notice that Orchestrator supports an utterance with multiple labels, so the intent variable cache is actually an array for holding multiple intent labels associated with the "current" utterance. Below is what happened after a user enter "greeting" as a new intent label using the 'i' commandlet. ``` Please enter a commandlet, "h" for help > i Please enter a "current" intent label > greeting > "Current" utterance: "hi" > "Current" intent label array: "greeting" > "New" intent label array: "" Please enter a commandlet, "h" for help > ``` #### Add example to Orchestrator core After a user enter "hi" and "greeting" during the interactive session, she/he can add this example to the Orchestrator core. It will become part of the example set Orchestrator core uses for predicting the intent for new queries. In a sense, this new example was added to a "training" set for a supervised machine learning model. The commandlet for adding a new example is 'a'. ``` Please enter a commandlet, "h" for help > a > Utterance 'hi' has been added to '[ "greeting" ]' > "Current" utterance: "hi" > "Current" intent label array: "greeting" > "New" intent label array: "" Please enter a commandlet, "h" for help > ``` #### Show statistics of the example set Once a user has entered a sufficient amount of intent/utterance examples into Orchestrator core, she/he can then use the 's' commandlet to show some statistics of the example set. ``` Please enter a commandlet, "h" for help > s > Per-label #examples: { "greeting": 1 } > Total #examples:1 > "Current" utterance: "hi" > "Current" intent label array: "greeting" > "New" intent label array: "" Please enter a commandlet, "h" for help > ``` #### Add examples to current cached intent label A user can enter more intent/utterance examples. Below is what happened if the user enters and added two more utterances ("hello" and "good morning") for the same "greeting" intents. ``` Please enter a commandlet, "h" for help > u Please enter an utterance > hello > "Current" utterance: "hello" > "Current" intent label array: "greeting" > "New" intent label array: "" Please enter a commandlet, "h" for help > a > Utterance 'hello' has been added to '[ "greeting" ]' > "Current" utterance: "hello" > "Current" intent label array: "greeting" > "New" intent label array: "" Please enter a commandlet, "h" for help > u Please enter an utterance > good morning > "Current" utterance: "good morning" > "Current" intent label array: "greeting" > "New" intent label array: "" Please enter a commandlet, "h" for help > a > Utterance 'good morning' has been added to '[ "greeting" ]' > "Current" utterance: "good morning" > "Current" intent label array: "greeting" > "New" intent label array: "" Please enter a commandlet, "h" for help > ``` #### Add example for a new intent label A user can also enter some utterances with different intents. Below is what happened if a user enters a new utterance "good bye" with a new intent "farewell." However the user forgot to clear the current intent labels, so the "good bye" intent is actually associated with two intents ("greeting,farewell"). Even though this is not completely wrong semantically, but a user can always clear the intent variable cache (with the 'ci' commandlet). Remember that the utterance and intents have not been added to Orchestrator core yet. ``` Please enter a commandlet, "h" for help > u Please enter an utterance > good bye > "Current" utterance: "good bye" > "Current" intent label array: "greeting" > "New" intent label array: "" Please enter a commandlet, "h" for help > i Please enter a "current" intent label > farewell > "Current" utterance: "good bye" > "Current" intent label array: "greeting,farewell" > "New" intent label array: "" Please enter a commandlet, "h" for help > ``` #### Clear intent label cache The commandlet to clear the intent label cache is 'ci'. Below is what happened if the user clear the intent label variable cache. Similarly, the utterance variable cache can be cleared by the 'cu' commandlet. ``` Please enter a commandlet, "h" for help > ci > "Current" utterance: "good bye" > "Current" intent label array: "" > "New" intent label array: "" Please enter a commandlet, "h" for help > ``` The user can then reenter the "farewell" intent into the cache and add it to Orchestrator core: ``` Please enter a commandlet, "h" for help > i Please enter a "current" intent label > farewell > "Current" utterance: "good bye" > "Current" intent label array: "farewell" > "New" intent label array: "" Please enter a commandlet, "h" for help > a > Utterance 'good bye' has been added to '[ "farewell" ]' > "Current" utterance: "good bye" > "Current" intent label array: "farewell" > "New" intent label array: "" Please enter a commandlet, "h" for help > ``` After a while, a user can re-issue the 's' commandlet and see the label utterance tabulation: From the session below, we can see that there are three utterances for the "greeting" intent, and one for the "farewell" intent. ``` Please enter a commandlet, "h" for help > s > Per-label #examples: { "greeting": 3, "farewell": 1 } > Total #examples:4 > "Current" utterance: "good bye" > "Current" intent label array: "farewell" > "New" intent label array: "" Please enter a commandlet, "h" for help > ``` #### Make prediction After there are some intent/utterance entered to Orchestrator core, the user can use Orchestrator core to make some prediction. Below is what happened if "bye" is entered as the "Current" utterance and use the 'p' commandlet for an prediction. Orchestrator core responded with a prediction of the "farewell" intent, a high score of 0.977, and the closest example within the "farewell" intent, "good bye". On the other hand, the "greeting" intent was predicted with a score of 0.691 and the closest example was "good morning." ``` Please enter an utterance > bye > "Current" utterance: "bye" > "Current" intent label array: "farewell" > "New" intent label array: "" Please enter a commandlet, "h" for help > p > Prediction: [ { "closest_text": "good bye", "score": 0.977452505191936, "label": { "name": "farewell", "label_type": 1, "span": { "length": 3, "offset": 0 } } }, { "closest_text": "good morning", "score": 0.6907451700060104, "label": { "name": "greeting", "label_type": 1, "span": { "length": 3, "offset": 0 } } } ] > "Current" utterance: "bye" > "Current" intent label array: "farewell" > "New" intent label array: "" Please enter a commandlet, "h" for help > ``` #### Hints on new intent label needed The same utterance, intent, add, predict commandlets can be repeated for a user to build a sizeable example set within Orchestrator core. A user try out some new utterance, such as "wake me up at 10AM" and he/she can see the prediction based on the two existing intent labels. As we can see for the prediction, the prediction scores are fairly low, 0.322 and 0.275. The low scores indicate that this new utterance may need a new intent label. ``` Please enter an utterance > wake me up at 10AM > "Current" utterance: "wake me up at 10AM" > "Current" intent label array: "farewell" > "New" intent label array: "" Please enter a commandlet, "h" for help > p > Prediction: [ { "closest_text": "good morning", "score": 0.322484918357831, "label": { "name": "greeting", "label_type": 1, "span": { "length": 18, "offset": 0 } } }, { "closest_text": "good bye", "score": 0.2750571664975262, "label": { "name": "farewell", "label_type": 1, "span": { "length": 18, "offset": 0 } } } ] > "Current" utterance: "wake me up at 10AM" > "Current" intent label array: "farewell" > "New" intent label array: "" Please enter a commandlet, "h" for help > ``` Below is what happened if the user enter a new intent label and add the "wake me up at 10AM" utterances to Ochestrator core. ``` Please enter a commandlet, "h" for help > ci > "Current" utterance: "wake me up at 10AM" > "Current" intent label array: "" > "New" intent label array: "" Please enter a commandlet, "h" for help > i Please enter a "current" intent label > alarm > "Current" utterance: "wake me up at 10AM" > "Current" intent label array: "alarm" > "New" intent label array: "" Please enter a commandlet, "h" for help > a > Utterance 'wake me up at 10AM' has been added to '[ "alarm" ]' > "Current" utterance: "wake me up at 10AM" > "Current" intent label array: "alarm" > "New" intent label array: "" Please enter a commandlet, "h" for help > ``` As we can see from issuing the 's' commandlet, now we have one more intents. ``` Please enter a commandlet, "h" for help > s > Per-label #examples: { "greeting": 3, "farewell": 1, "alarm": 1 } > Total #examples:5 > "Current" utterance: "wake me up at 10AM" > "Current" intent label array: "alarm" > "New" intent label array: "" Please enter a commandlet, "h" for help > ``` #### Evaluate the example set After a while, the user can use the 'v' commandlet to evaluate the example set. Below is what happened after issuing the 'v' commandlet. The user can open the evaluation report file and review the progress thus far. Since there are very few examples per intent, and the valuation is done in leave-one-out cross validation (LOOCV), so the report does not look good. ``` Please enter a commandlet, "h" for help > v > Leave-one-out cross validation is done and reports generated in 'experiment_evaluating_PrebuildDomain\orchestrator_predicting_set_summary.html' > "Current" utterance: "wake me up at 10AM" > "Current" intent label array: "alarm" > "New" intent label array: "" Please enter a commandlet, "h" for help > ``` #### Remove utterance However, if the user feels that the "wake me up at 10AM" is not what this chatbot is concerned with, she/he can remove the utterance using the 'r' commandlet. The 'r' commandlet call Orchestrator core to remove the intent and utterance example currently cached. On the other hand, the user can reference the evaluation report and pick out the utterance example and cache it into the "current" variables. Below is what happened that the user issue the 'vm' (validation/misclassified) commandlet and enter an index '2' for the No. 2 utterance and put it and its intent to the "Current" cache. Similarly a user can issue the 'va' (validation/ambiguous), 'vl' (validation/low-confidence), and the 'vd' (validation/duplicates) commands to pick up the utterance and intent examples and cache them into the variables. ``` Please enter a commandlet, "h" for help > vm Please enter an index from the Misclassified report > 2 > "Current" utterance: "wake me up at 10AM" > "Current" intent label array: "alarm" > "New" intent label array: "" Please enter a commandlet, "h" for help > ``` Below is what happened after issuing the 'r' commandlet to remove the "wake me up at 10AM" utterance. Since there is only one utterance for the "alarm" intent, the whole intent is eliminated. However, a user can simply issue the 'rl' commandlet to eliminate an intent label with all of its utterances. Now there are two intent left in the Orchestrator core. ``` > "Current" utterance: "wake me up at 10AM" > "Current" intent label array: "alarm" > "New" intent label array: "" Please enter a commandlet, "h" for help > r > Utterance 'wake me up at 10AM' has been removed from '[ "alarm" ]' > "Current" utterance: "wake me up at 10AM" > "Current" intent label array: "alarm" > "New" intent label array: "" Please enter a commandlet, "h" for help > s > Per-label #examples: { "greeting": 3, "farewell": 1 } > Total #examples:4 > "Current" utterance: "wake me up at 10AM" > "Current" intent label array: "alarm" > "New" intent label array: "" Please enter a commandlet, "h" for help > ``` #### Detailed configuration information For more detailed configuration information for Orchestrator, a user can issue the 'd' commandlet. Below is what happened after a user issue the 'd' commandlet, where there are several threshold used for the 'v' commandlet during creating an evaluation report. ``` Please enter a commandlet, "h" for help > d > Ambiguous closeness: 0.2 > Low-confidence closeness: 0.5 > Multi-label threshold: 1 > Unknown-label threshold: 0.3 > Obfuscation flag: false > Orchestrator configuration: { "CDF": "{\"zero\":0.400000,\"unknown\":0.700000,\"one\":1.000000}", "EmbedderVersion": 1, "Framework": "onnx", "Layers": 6, "MinRequiredCoreVersion": "1.0.0", "ModelFile": "D:\\testsBfCliOrchestrator\\_model\\model_bert_6l\\model.onnx", "ModelType": "dte_bert", "Name": "pretrained.20200924.microsoft.dte.00.06.en.onnx", "Publisher": "Microsoft", "Version": "pretrained.20200924.microsoft.dte.00.06.en.onnx", "VocabFile": "D:\\testsBfCliOrchestrator\\_model\\model_bert_6l\\vocab.txt", "ignore_same_example": true, "knn_k": 3, "nlr_path": "D:\\testsBfCliOrchestrator\\_model\\model_bert_6l", "use_compact_embeddings": true, "use_gap": false, "use_unknown": true } > Current label-index map: { "farewell": 0, "greeting": 1 } > "Current" utterance: "wake me up at 10AM" > "Current" intent label array: "alarm" > "New" intent label array: "" Please enter a commandlet, "h" for help > ``` - The "Ambiguous closeness" threshold is currently set to 0.2, which means if an utterance is correctly predicted, but there are some other intent with a score close to 80% of the ground-truth intent label, then that close utterance will be listed as an "ambiguous" prediction in the evaluation report. - The "Low-confidence" threshold is default to 0.5, which means that if an utterance's ground-truth intent label is predicted to have the highest score, but the score is lower than 0.5, then this utterance will be listed as "low confidence" - The "Unknown-label" threshold is default to 0.3, which means that if the highest predicted intent score is lower than 0.3, then Orchestrator will simply declare that the intent is unknown. - A user can use the 'vo' commandlet to set the "obfuscate" flag, so that the 'v' commandlet can create an evaluation report with all the intent and utterances obfuscated. An obfuscated report only show some metadata of the example set without any intent or utterance disclosed, so that the report might be shareable for benchmark and comparison purpose. The 'd' commandlet also list some internal information of the base model and settings used by Orchestrator core. These information can be used for debugging purpose. These thresholds can be reset using the 'vat', 'vlt', 'vmt' and 'vut' commandlets. #### Create a new snapshot (.BLU) file After a user has reached a milestone, she/he can create a new snapshot (.BLU) file using the 'n' command. In the future, the user can run an interactive session preloaded with this new snapshot file and keep refining the intent and utterance examples. Below is what happened after the running the 'n' command and the location of the new snapshot file. ``` Please enter a commandlet, "h" for help > n > A new snapshot has been saved to 'experiment_evaluating_PrebuildDomain\orchestrator_predicting_snapshot_set.blu' > "Current" utterance: "wake me up at 10AM" > "Current" intent label array: "alarm" > "New" intent label array: "" Please enter a commandlet, "h" for help > ``` #### Exit the session At the end, a user can enter the 'q' commandlet to exit the session. ### Start an interactive session with a training set Since there is already a .blu snapshot file from the last session, a user can use the interactive command to continue refining the example set by using the "--in" argument. #### Argument setup ``` > set ORCHESTRATOR_MODEL= > set EVALUATING_OUTPUT= > bf orchestrator:interactive --out=%EVALUATING_OUTPUT% --model=%ORCHESTRATOR_MODEL% --in=%EVALUATING_OUTPUT%\orchestrator_predicting_snapshot_set.blu ``` #### Show statistics of the example set After a new interactive session, the user can issue the 's' commandlet and take a look of the intent/utterance distribution. ``` > "Current" utterance: "" > "Current" intent label array: "" > "New" intent label array: "" Please enter a commandlet, "h" for help > s > Per-label #examples: { "greeting": 3, "farewell": 1 } > Total #examples:4 > "Current" utterance: "" > "Current" intent label array: "" > "New" intent label array: "" Please enter a commandlet, "h" for help > ``` #### Make predictions on utterances The user can try out some more utterances. ``` Please enter a commandlet, "h" for help > u Please enter an utterance > bye bye > "Current" utterance: "bye bye" > "Current" intent label array: "" > "New" intent label array: "" Please enter a commandlet, "h" for help > p > Prediction: [ { "closest_text": "good bye", "score": 0.9852574345531401, "label": { "name": "farewell", "label_type": 1, "span": { "length": 7, "offset": 0 } } }, { "closest_text": "good morning", "score": 0.7110401992969432, "label": { "name": "greeting", "label_type": 1, "span": { "length": 7, "offset": 0 } } } ] > "Current" utterance: "bye bye" > "Current" intent label array: "" > "New" intent label array: "" Please enter a commandlet, "h" for help > ``` #### Add examples to existing language model The predicted intent label was "farewell" with a very high score of 0.985. However, suppose it's a low score, then the user can improve model by adding the utterance into the example set. Below is what happened after the "bye bye" utterance is added. ``` Please enter a commandlet, "h" for help > i Please enter a "current" intent label > farewell > "Current" utterance: "bye bye" > "Current" intent label array: "farewell" > "New" intent label array: "" Please enter a commandlet, "h" for help > a > Utterance 'bye bye' has been added to '[ "farewell" ]' > "Current" utterance: "bye bye" > "Current" intent label array: "farewell" > "New" intent label array: "" Please enter a commandlet, "h" for help > s > Per-label #examples: { "greeting": 3, "farewell": 2 } > Total #examples:5 > "Current" utterance: "bye bye" > "Current" intent label array: "farewell" > "New" intent label array: "" Please enter a commandlet, "h" for help > ``` Below is what happened when a user enters a new utterance "good day" and makes a prediction of it. As we can see that the prediction scores for the "greeting" and "farewell" intents are both very high and close. The user then added the "good day" utterance to the "farewell" intent. ``` Please enter a commandlet, "h" for help > u Please enter an utterance > good day > "Current" utterance: "good day" > "Current" intent label array: "farewell" > "New" intent label array: "" Please enter a commandlet, "h" for help > p > Prediction: [ { "closest_text": "good morning", "score": 0.8536692972984349, "label": { "name": "greeting", "label_type": 1, "span": { "length": 8, "offset": 0 } } }, { "closest_text": "good bye", "score": 0.8114553301042285, "label": { "name": "farewell", "label_type": 1, "span": { "length": 8, "offset": 0 } } } ] > "Current" utterance: "good day" > "Current" intent label array: "farewell" > "New" intent label array: "" Please enter a commandlet, "h" for help > a > Utterance 'good day' has been added to '[ "farewell" ]' > "Current" utterance: "good day" > "Current" intent label array: "farewell" > "New" intent label array: "" Please enter a commandlet, "h" for help > s > Per-label #examples: { "greeting": 3, "farewell": 3 } > Total #examples:6 > "Current" utterance: "good day" > "Current" intent label array: "farewell" > "New" intent label array: "" Please enter a commandlet, "h" for help > ``` #### Relabel utterances However, the user may later decide to change the intent for "good day" from "farewell" to "greeting". She/he can them use the 'ni' commandlet to enter "greeting" into the "New" intent label cache. After that, the user can issue the 'c' commandlet to change the intent label for "good day". As we can see below, the label/utterance distribution has changed afterward. Notice that the user can use the 'cni' commandlet to clear the "New" intent label cache. ``` Please enter a commandlet, "h" for help > s > Per-label #examples: { "greeting": 3, "farewell": 3 } > Total #examples:6 > "Current" utterance: "good day" > "Current" intent label array: "farewell" > "New" intent label array: "" Please enter a commandlet, "h" for help > ni Please enter a "new" intent label > greeting > "Current" utterance: "good day" > "Current" intent label array: "farewell" > "New" intent label array: "greeting" Please enter a commandlet, "h" for help > c > Utterance 'good day' has been moved from '[ "farewell" ]' to '[ "greeting" ]' > "Current" utterance: "good day" > "Current" intent label array: "farewell" > "New" intent label array: "greeting" Please enter a commandlet, "h" for help > s > Per-label #examples: { "greeting": 4, "farewell": 2 } > Total #examples:6 > "Current" utterance: "good day" > "Current" intent label array: "farewell" > "New" intent label array: "greeting" Please enter a commandlet, "h" for help > ``` #### Check existence of an utterance in the language model Also, the user can enter an utterance and see if it's in the Orchestrator Core's example set or not using the 'f' commandlet. Below is what happened when a user tries to find out if "good evening" is in the example set of not. ``` Please enter a commandlet, "h" for help > u Please enter an utterance > good evening > "Current" utterance: "good evening" > "Current" intent label array: "farewell" > "New" intent label array: "greeting" Please enter a commandlet, "h" for help > f > The "current" utterance 'good evening' is not in the model. > "Current" utterance: "good evening" > "Current" intent label array: "farewell" > "New" intent label array: "greeting" Please enter a commandlet, "h" for help > ``` ## List of interactive commandlets At any time, if a user needs to reference help messages for the many commandlets, he/she can issue the 'h' command for a brief help page on all the commandlets. Below is the list of the commandlets that can be issued during a 'interactive' session. ### 'h' - help command h - print the help messages with short description of each command ### 'q' - quit command q - quit ### 'd' - display command d - display the utterance, intent label array input caches, Orchestrator config, and the label-index map ### 's' - statistics command s - show label-utterance statistics of the model examples ### 'u' - utterance command u - enter a new utterance and save it as the "current" utterance input ### 'cu' - clear utterance variable command cu - clear the "current" utterance input cache ### 'i' - intent variable command i - enter an intent and add it to the "current" intent label array input (can be an index for retrieving a label from the label-index map) ### 'ci' - clear intent variable command ci - clear the "current" intent label array input cache ### 'ni' - new intent command ni - enter an intent and add it to the "new" intent label array input (can be an index for retrieving a label from the label-index map) ### 'cni' - clear new intent command cni - clear the "new" intent label array input cache ### 'f' - find command f - find the "current" utterance if it is already in the model example set ### 'p' - predict command p - make a prediction on the "current" utterance input ### 'v' - validate command v - validate the model and save analyses (validation report) to "experiment_predicting_va\orchestrator_predicting_set_summary.html" ### 'vd' - utterance/labels retrieval command (from the Duplicates section of the validation report) vd - reference a validation Duplicates report (previously generated by the "v" command) and enter an index for retrieving utterance/intents into "current" ### 'va' - utterance/labels retrieval command (from the Ambiguous section of the validation report) va - reference a validation Ambiguous report (previously generated by the "v" command) and enter an index for retrieving utterance/intents into "current" ### 'vm' - utterance/labels retrieval command (from the Misclassified section of the validation report) vm - reference a validation Misclassified report and enter an index (previously generated by the "v" command) for retrieving utterance/intents into "current" ### 'vl' - utterance/labels retrieval command (from the Low Confidence section of the validation report) vl - reference a validation LowConfidence report (previously generated by the "v" command) and enter an index for retrieving utterance/intents into "current" ### 'vat' - ambiguous threshold command vat - enter a new validation-report ambiguous closeness threshold ### 'vlt' - low-confidence threshold command vlt - enter a new validation-report low-confidence threshold ### 'vmt' - multi-label threshold command vmt - enter a new multi-label threshold ### 'vut' - unknown label threshold command vut - enter a new unknown-label threshold ### 'vo' - obfuscate command vo - enter a boolean for obfuscating labels/utterances or not in evaluation reports generated by the "v" command' ### 'a' - add command a - add the "current" utterance and intent labels to the model example set ### 'r' - remove command r - remove the "current" utterance and intent labels from the model example set ### 'c' - change command c - remove the "current" utterance's intent labels and then add it with the "new" intent labels to the model example set ### 'rl' - remove label command rl - remove the "current" intent labels from the model example set ### 'n' - new snapshot command n - create a new snapshot of model examples and save it to "experiment_predicting_va\orchestrator_predicting_training_set.blu" ## References - [BF Orchestrator CLI](https://aka.ms/bforchestratorcli) [1]:https://aka.ms/bforchestratorcli "BF Orchestrator CLI" ================================================ FILE: Orchestrator/docs/BFOrchestratorReport.md ================================================ # Report Interpretation The [BF Orchestrator CLI][1] has a "test" command for evaluating the performance of an Orchestrator snapshot file (with .blu extension). A snapshot is composed of natural language representation base model (see [models][3]) along with a set of examples provided in a label file (typically a [.lu file][4]). The snapshot file is used in Bot Framework to detect intents from user utterances. In order to achieve high quality natural language processing (e.g. intent detection), it is necessary to assess and refine the quality of the model. This is much simplified in Orchestrator because of its use of pre-trained models. The optimization cycle is required in order to account for human language variations. Learn more about Machine Learning evaluation methodology in the [References](# references) section below. To interpret the report, use the following guidance to actions to improve the snapshot file: - Incase utterances are the same or semantically similar, mrerge the two intent labels. - If the utterances are not semantically similar, split an intent's utterance pool and create a new intent label. - It the utterance are semantically close to a different intent label, change the intent label. - Rephrase an utterance and make it semantically closer to other utterances labled with the same intent. - Add more utterances to an intent label as the intent's utterance pool could be too scarce. - Remove some utterances from an intent label if there were too many utterances label to it. # Report Organization The test command thus produces a folder with HTML report and a few supporting artifacts as follows: - orchestrator_testing_set_ground_truth_instances.json: test instance ground-truth file in JSON format. - orchestrator_testing_set_labels.txt: intent labels in a plain text file. - orchestrator_testing_set_prediction_instances.json: test instance prediction file in JSON format. - orchestrator_testing_set_scores.txt: test instance prediction file in a plain TSV format. - orchestrator_testing_set_summary.html: report summary in HTML format The report summary contains several sections as follows: ## Intent / Utterance Statistics This section contains label and utterance distributions. It has two statistical sections, one for labels, the other utterances. Attached is an example rendition of the section. - Label statistics - Utterance statistics ![Evaluation Report Intent/Utterance Statistics](media/EvaluationReportTabVaIntentUtteranceStatistics.png) ### Label statistics Label statistics lists the number of utterances labeled to each label. Additional metrics include utterance prevalence (ratio) for every label. The distributions can give Orchestrator users an overall view of the labels and utterances, and whether the distributions are skewed and emphasize too much on some labels, but not others. A machine learn model may learn more from a label (intent) with more instances (utterances) labeled to it. Thus, a developer can check this table and see if some intents needs more utterances in the snapshot file. ### Utterance statistics On the other hand, utterance statistics focus on the #label distribution by the utterances. Some utterances are labeled with more than one intents, which might not be desirable and could be a bug. This table reflects the distribution of multi-label utterances. From the above screen snapshot, we can see that there are two utterances labeled twice with distinct labels. Those multi-label utterances will be listed in the next section and the owner can decide to remove the duplicates from the snapshot file. ## Utterance Duplicates Sometimes some dataset might contain utterances tagged with the same labels multiple times. The report also lists this redundancy. This section has two sub-sections: - Multi-label utterances and their labels - Duplicate utterance and label pairs They report on utterances with duplicate or multiple labels. A duplicate utterance is detected when it is present more than once in a snapshot file. Sometimes some dataset might even contain utterances tagged with the same labels multiple times. The report also lists the redundancy of label/utterance pairs. Orchestrator will deduplicate such redundancy. Please see the attached screen snapshot as an example. ![Evaluation Report Utterance Duplicates](media/EvaluationReportTabVaUtteranceDuplicates.png) ## Ambiguous This section reports on utterances with ambiguous predictions. For an evaluation utterance, if an Orchestrator model correctly predicts its intent label, then it's a true positive instance. Every intent label will be predicted with a score, which is essentially the probability or confidence for that label prediction. The predicted intent usually is the one with the highest score. If the Orchestrator model also makes some other high-score prediction close to that of the correctly predicted label, then we call such a prediction "ambiguous." In this section, the report lists all the utterances with an ambiguous prediction in a table. The table has several columns: - Utterance -- the utterance - Labels -- the true labels for the utterance - Predictions -- the labels predicted by the Orchestrator model - Close Predictions -- some other labels predicted with a close, high score to that of the predicted label. Besides the prediction score, the report also shows the closest example to the utterance within the label's utterance set. Below is a screen snapshot of an ambiguous report: ![Evaluation Report Ambiguous](media/EvaluationReportTabVaAmbiguous.png) Ambiguous utterances can be a sign of overlapping intent labels. In another word, two intent labels may have their utterance pools semantically close to each other. In the example report about, the utterance "what my events today" was correctly predicted with the Calendar intent label and the closest example is "What is on my calendar today". However the FAQ3 intent was also predicted with a high school and the closest example from that intent is "What's going on today." From these two closest example, one can remove the latter from the FAQ3 intent as it is now more specific to the Calendar intent, then just a FAQ example. ## Misclassified This section reports on utterances with incorrect predictions. A misclassified predication is one where an Orchestrator model falsely predicts its intent label. Usually the label with the highest prediction score is chosen as the predicted label, but it can be different from the ground-truth label for the utterance. Similar to the last section, the report also lists the prediction and ground-truth labels with their prediction scores and closest examples. Below is a screen snapshot of the misclassified report. A user can also follow the utterances list and decide to update their intent labels and or rephrase the utterances themselves. ![Evaluation Report Misclassified](media/EvaluationReportTabVaMisclassified.png) ## Low Confidence This section reports on predictions that scored too low to be considered "confident" intent detection. Sometimes a prediction may be predicted correctly with the highest scores among all labels, but the score is very low, lower than a threshold. We call such predictions low confidence. Notice that there are several default thresholds used to guide producing the report sections thus far. Usually an Orchestrator user can just predict a label with the highest score, but he/she can also utilize some thresholds for a fine-tuned prediction in a chat bot's dialog logic. Just like the last sections, the report lists the prediction and ground-truth labels with their prediction scores and closest examples. Also like the previous sections, the utterances listed in this section can guide a user how to improve the snapshot file. ![Evaluation Report Low Confidence](media/EvaluationReportTabVaLowConfidence.png) ## Metrics The Metrics section is an advanced report that contains analytics that is common in Machine Learning evaluation methodologies. Advanced machine-learning practitioners may analyze the overall model performance expressed in machine learning metrics. In this section, the report calculates some common metrics in two sections: - Confusion matrix metrics - Average confusion matrix metrics ![Evaluation Report Metrics](media/EvaluationReportTabVaMetrics.png) ### Confusion matrix metrics In this table, the Orchestrator CLI test command reads an evaluation set with ground-truth labels. An evaluation set contains a collection of utterances and their labels. It then calls the Orchestrator base model and makes a prediction for every utterance in the set and generate predicted labels for every utterance. It then compares the predicted labels against the ground-truth labels and creates a table of per-label binary confusion matrices. For a binary confusion matrix, there are four cells: - #True Positives (TP) - #False Positives (FP) - #False Neagatives (FN) - #True Negatives (TN) For a label, if it exists in an utterance's ground-truth label set as well as in the predicted label set, then that utterance is a TP for the label. If the label only exists in the utterance's predicted label set, then it's a false positive. If the label only exists in the utterance's ground-truth set, then it's a false negative. If the label does not exist in either the ground-truth or predicted set, then it's a true negative. By the way, for entity extraction, as a label contains an entity name, entity offset in an utterance and its length, there can be numerous true negatives, since there are limitless combinations of the entity attributes not in the ground-truth or the predicted sets. Thus, for intent prediction evaluation, all four cells are used to calculate confusion matrix metrics, but only the first three are used for entity-extraction evaluation. Using just the first three cells of a binary confusion matrix, the Orchestrator "test" command can then calculate some more sophisticated metrics, including - Precision - TP / (TP + FP) - Recall - TP / (TP + FN) - F1 - harmonic mean of precision and recall These three metrics do not use TN, but the simple accuracy metric need all 4 cells, including TN. For details of many confusion matrix metrics, please reference [Wikipedia: Confusion matrix][2]. ### Average confusion matrix metrics Since Orchestrator can evaluate multiple labels, one confusion matrix for each, there can be many metrics for detailed analysis. For reporting and KPI purpose, it would be great to aggregate all these metrics for an overall metric and model performance. There are many nuanced ways to aggregate confusion matrix metrics. For comparing models, it's critical to compare based on a consistent formula. Please reference the [BF Orchestrator CLI][1] readme page for advanced CLI usage details. ## Thresholds This evaluation report is created using several thresholds that they can also be useful in building chat bot logic. These thresholds can be reset through environment variables listed below: - ambiguousClosenessThreshold: default to 0.2, which means that if there are labels predicted to have a score close to within 20% of the top and corrected predicted label, then this utterance is ambiguously predicted. - lowConfidenceScoreThreshold: default to 0.5, which means that if the top predicted score is lower than 0.5, then the prediction is considered low confidence. - multiLabelPredictionThreshold: default to 1, which means that the report will predict only one label. However, if the threshold is lower than 1, then every label with a predicted score higher than that will be predicted. This threshold is usually used for multi-label, multi-intent scenarios. - unknownLabelPredictionThreshold:default to 0.3, which means that the evaluation process will consider a prediction UNKNOWN if the score is lower than that threshold. ## References - [BF Orchestrator CLI](https://aka.ms/bforchestratorcli) - [Wikipedia: Confusion matrix](https://en.wikipedia.org/wiki/Confusion_matrix) - [Wikipedia: Training, validation, and test sets](https://en.wikipedia.org/wiki/Training,_validation,_and_test_sets) - [Machine Learning Mastery](https://machinelearningmastery.com/difference-test-validation-datasets/). [1]:https://aka.ms/bforchestratorcli "BF Orchestrator CLI" [2]:https://en.wikipedia.org/wiki/Confusion_matrix "Wikipedia: Confusion matrix" [3]:https://aka.ms/nlrmodels "NLR Models" [4]:https://docs.microsoft.com/en-us/azure/bot-service/file-format/bot-builder-lu-file-format?view=azure-bot-service-4.0 ".LU format" [5]: https://github.com/microsoft/botframework-cli/tree/main/packages/orchestrator#bf-orchestratortest "bf orchestrator:test" ================================================ FILE: Orchestrator/docs/BFOrchestratorUsage.md ================================================ # BF Orchestrator Command Usage [BF Orchestrator Command ][1] is used to manage Orchestrator language understanding assets - retrieve, create, evaluate, and tune Orchestrator models to be used in bot development. Typical use is for intent routing to the appropriate language service, LUIS or QnA Maker for additional language tasks such as entity extraction or answer retrieval. This use was previously enabled by use of the legacy *dispatch* CLI. In addition, intent routing can be used to delegate conversations to a backend bot skill. Entity recognition is on the roadmap as a future capability. ## Primary Scenario The mainstream bot language recognition development cycle with Orchestrator is generally as follows: 1. Create Intent label / utterance examples file in [.lu format][2] (will be referred to as the *label file*). 2. Download base model. 3. Combine the label file .lu from (1) with the base model from (2) to create a .blu file (will be referred to as the *snapshot* file). 4. Test and refine quality of utterance to intent recognition. 5. Integrate Orchestrator language recognizer in your bot. ## Steps We will use the primary workflow to illustrate how to use Orchestrator commands for full development cycle. ### 1. Create Intent label / utterance examples file in [.lu format][2] If you are developing a new language model, simply refer to [Language understanding][8] documentation to author label files. Depending on how you plan to design your bot, you may use a single label file corresponding to a single snapshot file for the entire bot or multiple pairs, each for every adaptive dialog where you plan to use Orchestrator as a recognizer. In case of **migration from legacy dispatch**, simply add LUIS app(s) or QnAMaker kb(s) as Orchestrator data sources. Refer to the Dispatch migration documentation [here][11] for more information. Example: ``` > bf orchestrator:add -t luis --id --endpoint -k -v --routingName l_ > bf orchestrator:add -t qna --id -k --routingName q_ ``` ### 2. Download base model Create a new folder, say *models*, and download the default base model using: ``` bf orchestrator:basemodel:get --out ./models ``` out parameter is optional. If not specified, base model files will be downloaded to the current working directory. See also the ```orchestrator:basemodel:list``` command if you wish to download and experiment with different base models. (see descriptions [here][4] ). ### 3. Combine the label file .lu from (1) with the base model from (2) to create a .blu file There are two ways to create Orchestrator snapshot file(s), depending on the usage scenarios: - **To route utterances to LUIS/QnA language services** Use ```bf orchestrator:create``` command to combine the base model with the label file(s) to create snapshot file for use by the orchestrator recognizer. If using a single folder, create it prior, say *generated*, and specify it in --out parameter: ``` bf orchestrator:create --model --in