Repository: joashc/HaxlSharp Branch: master Commit: 196d6db816d6 Files: 55 Total size: 151.6 KB Directory structure: gitextract_d6xprphf/ ├── .gitignore ├── CONTRIBUTORS.md ├── HaxlSharp.Core/ │ ├── BlockedRequest.cs │ ├── CacheKeyGenerator.cs │ ├── Fetch.cs │ ├── Fetcher.cs │ ├── HaxlLogEntry.cs │ ├── HaxlSharp.Core.csproj │ ├── HaxlSharp.Core.nuspec │ ├── Internal/ │ │ ├── Applicative/ │ │ │ ├── HaxlApplicative.cs │ │ │ └── SplitApplicative.cs │ │ ├── Base/ │ │ │ ├── Base.cs │ │ │ ├── ByteString.cs │ │ │ ├── Func.cs │ │ │ └── HaxlConstants.cs │ │ ├── Expressions/ │ │ │ ├── LetExpression.cs │ │ │ ├── ParameterAccessVisitor.cs │ │ │ ├── ParseExpression.cs │ │ │ └── RebindToScope.cs │ │ ├── Haxl.cs │ │ ├── HaxlCache.cs │ │ ├── Result.cs │ │ ├── RunFetch.cs │ │ ├── Scope.cs │ │ └── Types/ │ │ ├── ApplicativeGroup.cs │ │ ├── BindProjectPair.cs │ │ ├── BoundExpression.cs │ │ ├── CacheResult.cs │ │ ├── ExpressionVariables.cs │ │ ├── FreeVariable.cs │ │ ├── QueryStatement.cs │ │ ├── ShowList.cs │ │ ├── Statement.cs │ │ └── Unit.cs │ ├── Properties/ │ │ └── AssemblyInfo.cs │ ├── Response.cs │ └── Returns.cs ├── HaxlSharp.Fetcher/ │ ├── FetcherBuilder.cs │ ├── HashedRequestKey.cs │ ├── HaxlFetcher.cs │ ├── HaxlSharp.Fetcher.csproj │ ├── HaxlSharp.Fetcher.nuspec │ ├── Properties/ │ │ └── AssemblyInfo.cs │ └── packages.config ├── HaxlSharp.Test/ │ ├── ApplicativeRewriteTest.cs │ ├── BindExpressionParseTest.cs │ ├── Blog.cs │ ├── ExpressionTests.cs │ ├── HaxlSharp.Test.csproj │ ├── MockData.cs │ └── Properties/ │ └── AssemblyInfo.cs ├── HaxlSharp.sln ├── LICENCE ├── README.md └── buildNuget.bat ================================================ FILE CONTENTS ================================================ ================================================ FILE: .gitignore ================================================ # Created by https://www.gitignore.io/api/visualstudio nuget.exe nuget/ ### VisualStudio ### ## Ignore Visual Studio temporary files, build results, and ## files generated by popular Visual Studio add-ons. # User-specific files *.suo *.user *.userosscache *.sln.docstates # User-specific files (MonoDevelop/Xamarin Studio) *.userprefs # Build results [Dd]ebug/ [Dd]ebugPublic/ [Rr]elease/ [Rr]eleases/ x64/ x86/ bld/ [Bb]in/ [Oo]bj/ [Ll]og/ # Visual Studio 2015 cache/options directory .vs/ # Uncomment if you have tasks that create the project's static files in wwwroot #wwwroot/ # MSTest test Results [Tt]est[Rr]esult*/ [Bb]uild[Ll]og.* # NUNIT *.VisualState.xml TestResult.xml # Build Results of an ATL Project [Dd]ebugPS/ [Rr]eleasePS/ dlldata.c # DNX project.lock.json artifacts/ *_i.c *_p.c *_i.h *.ilk *.meta *.obj *.pch *.pdb *.pgc *.pgd *.rsp *.sbr *.tlb *.tli *.tlh *.tmp *.tmp_proj *.log *.vspscc *.vssscc .builds *.pidb *.svclog *.scc # Chutzpah Test files _Chutzpah* # Visual C++ cache files ipch/ *.aps *.ncb *.opendb *.opensdf *.sdf *.cachefile *.VC.db *.VC.VC.opendb # Visual Studio profiler *.psess *.vsp *.vspx *.sap # TFS 2012 Local Workspace $tf/ # Guidance Automation Toolkit *.gpState # ReSharper is a .NET coding add-in _ReSharper*/ *.[Rr]e[Ss]harper *.DotSettings.user # JustCode is a .NET coding add-in .JustCode # TeamCity is a build add-in _TeamCity* # DotCover is a Code Coverage Tool *.dotCover # NCrunch _NCrunch_* .*crunch*.local.xml nCrunchTemp_* # MightyMoose *.mm.* AutoTest.Net/ # Web workbench (sass) .sass-cache/ # Installshield output folder [Ee]xpress/ # DocProject is a documentation generator add-in DocProject/buildhelp/ DocProject/Help/*.HxT DocProject/Help/*.HxC DocProject/Help/*.hhc DocProject/Help/*.hhk DocProject/Help/*.hhp DocProject/Help/Html2 DocProject/Help/html # Click-Once directory publish/ # Publish Web Output *.[Pp]ublish.xml *.azurePubxml # TODO: Comment the next line if you want to checkin your web deploy settings # but database connection strings (with potential passwords) will be unencrypted *.pubxml *.publishproj # Microsoft Azure Web App publish settings. Comment the next line if you want to # checkin your Azure Web App publish settings, but sensitive information contained # in these scripts will be unencrypted PublishScripts/ # NuGet Packages *.nupkg # The packages folder can be ignored because of Package Restore **/packages/* # except build/, which is used as an MSBuild target. !**/packages/build/ # Uncomment if necessary however generally it will be regenerated when needed #!**/packages/repositories.config # NuGet v3's project.json files produces more ignoreable files *.nuget.props *.nuget.targets # Microsoft Azure Build Output csx/ *.build.csdef # Microsoft Azure Emulator ecf/ rcf/ # Windows Store app package directories and files AppPackages/ BundleArtifacts/ Package.StoreAssociation.xml _pkginfo.txt # Visual Studio cache files # files ending in .cache can be ignored *.[Cc]ache # but keep track of directories ending in .cache !*.[Cc]ache/ # Others ClientBin/ ~$* *~ *.dbmdl *.dbproj.schemaview *.pfx *.publishsettings node_modules/ orleans.codegen.cs # Since there are multiple workflows, uncomment next line to ignore bower_components # (https://github.com/github/gitignore/pull/1529#issuecomment-104372622) #bower_components/ # RIA/Silverlight projects Generated_Code/ # Backup & report files from converting an old project file # to a newer Visual Studio version. Backup files are not needed, # because we have git ;-) _UpgradeReport_Files/ Backup*/ UpgradeLog*.XML UpgradeLog*.htm # SQL Server files *.mdf *.ldf # Business Intelligence projects *.rdl.data *.bim.layout *.bim_*.settings # Microsoft Fakes FakesAssemblies/ # GhostDoc plugin setting file *.GhostDoc.xml # Node.js Tools for Visual Studio .ntvs_analysis.dat # Visual Studio 6 build log *.plg # Visual Studio 6 workspace options file *.opt # Visual Studio LightSwitch build output **/*.HTMLClient/GeneratedArtifacts **/*.DesktopClient/GeneratedArtifacts **/*.DesktopClient/ModelManifest.xml **/*.Server/GeneratedArtifacts **/*.Server/ModelManifest.xml _Pvt_Extensions # Paket dependency manager .paket/paket.exe paket-files/ # FAKE - F# Make .fake/ # JetBrains Rider .idea/ *.sln.iml ================================================ FILE: CONTRIBUTORS.md ================================================ # Contributors * [Joash Chong](https://github.com/joashc) - Author * [Courtney Strachan](https://github.com/cstrachan88) ================================================ FILE: HaxlSharp.Core/BlockedRequest.cs ================================================ using System; using System.Threading.Tasks; namespace HaxlSharp { /// /// A request that's blocking the completion of a fetch. /// /// /// We simulate existential types by packaging the request with its type information. /// public class BlockedRequest { public readonly object TypedRequest; public readonly Type RequestType; public readonly string BindName; public readonly TaskCompletionSource Resolver; public BlockedRequest(object typedRequest, Type requestType, string bindName) { TypedRequest = typedRequest; RequestType = requestType; BindName = bindName; Resolver = new TaskCompletionSource(); } } } ================================================ FILE: HaxlSharp.Core/CacheKeyGenerator.cs ================================================ using System; using System.Collections.Generic; using System.Linq; using System.Text; using System.Threading.Tasks; namespace HaxlSharp { /// /// Generates a unique key per request. /// public interface CacheKeyGenerator { string ForRequest(Returns request); } } ================================================ FILE: HaxlSharp.Core/Fetch.cs ================================================ using System; using System.Collections.Generic; using System.ComponentModel; using System.Linq; using System.Linq.Expressions; using System.Threading.Tasks; using static HaxlSharp.Internal.Base; using HaxlSharp.Internal; using System.Diagnostics; namespace HaxlSharp { /// /// Fetch a result. /// /// /// This is a free monad that leaves its expression tree open for inspection. /// public interface Fetch : Fetchable { [EditorBrowsable(EditorBrowsableState.Never)] IEnumerable CollectedExpressions { get; } [EditorBrowsable(EditorBrowsableState.Never)] LambdaExpression Initial { get; } } public interface Fetchable { [EditorBrowsable(EditorBrowsableState.Never)] Haxl ToHaxlFetch(string bindTo, Scope scope); } /// /// Monadic bind that just collects the query expression tree for inspection. /// [EditorBrowsable(EditorBrowsableState.Never)] public class Bind : Fetch { public IEnumerable CollectedExpressions { get; } public readonly Fetch Fetch; public bool IsLet; public Bind(IEnumerable binds, Fetch expr) { CollectedExpressions = binds; Fetch = expr; } public LambdaExpression Initial => Fetch.Initial; public Haxl ToHaxlFetch(string bindTo, Scope scope) { var bindSplit = SplitApplicative.SplitBind(CollectedExpressions, Initial); var newScope = IsLet ? scope : new Scope(scope); return HaxlApplicative.ToFetch(bindSplit, bindTo, newScope); } } /// /// All binds must terminate in a FetchNode. /// public abstract class FetchNode : Fetch { private static readonly IEnumerable emptyList = new List(); public IEnumerable CollectedExpressions => emptyList; public LambdaExpression Initial => Expression.Lambda(Expression.Constant(this)); public abstract Haxl ToHaxlFetch(string bindTo, Scope scope); } /// /// Wraps a primitive request type. /// [EditorBrowsable(EditorBrowsableState.Never)] public class Request : FetchNode { public readonly Returns request; public Request(Returns request) { this.request = request; } public object WarnIfNull(object result, Action logger) { if (result == null) logger(Warn($"The request type '{request.GetType().Name}' returned a null.")); return result; } public override Haxl ToHaxlFetch(string bindTo, Scope scope) { Func, Haxl> DoneFromTask = t => Haxl.FromFunc((c, l) => Done.New(_ => scope.Add(bindTo, WarnIfNull(t.Result, l)))); return Haxl.FromFunc((cache, logger) => { var cacheResult = cache.Lookup(request); return cacheResult.Match ( notFound => { var blocked = new BlockedRequest(request, request.GetType(), bindTo); cache.Insert(request, blocked); return Blocked.New( new List { blocked }, DoneFromTask(blocked.Resolver.Task) ); }, found => { var task = found.Resolver.Task; if (task.IsCompleted) return Done.New(_ => { var result = task.Result; return scope.Add(bindTo, WarnIfNull(result, logger)); }); return Blocked.New( new List(), DoneFromTask(task) ); } ); }); } public Type RequestType => request.GetType(); } /// /// Applicative sequence. /// [EditorBrowsable(EditorBrowsableState.Never)] public class RequestSequence : FetchNode> { public readonly IEnumerable List; public readonly Func> Bind; public RequestSequence(IEnumerable list, Func> bind) { List = list; Bind = bind; } public override Haxl ToHaxlFetch(string bindTo, Scope parentScope) { var childScope = new Scope(parentScope); var binds = List.Select(Bind).ToList(); var fetches = binds.Select((f, i) => f.ToHaxlFetch($"{bindTo}[{i}]", childScope)).ToList(); var concurrent = fetches.Aggregate((f1, f2) => f1.Applicative(f2)); return concurrent.Bind(scope => Haxl.FromFunc((cache, logger) => Done.New(_ => { var values = scope.ShallowValues.Select(v => (B)v).ToList(); return scope.WriteParent(bindTo, values); } ))); } } /// /// Wraps the value in a Fetch monad. /// [EditorBrowsable(EditorBrowsableState.Never)] public class FetchResult : FetchNode { public readonly A Value; public FetchResult(A value) { Value = value; } public override Haxl ToHaxlFetch(string bindTo, Scope scope) { return Haxl.FromFunc((cache, logger) => Done.New(_ => scope.Add(bindTo, Value))); } } /// /// Maps the result of the fetch with given function. /// [EditorBrowsable(EditorBrowsableState.Never)] public class Select : FetchNode { public readonly Fetch Fetch; public readonly Expression> Map; public Select(Fetch fetch, Expression> map) { Fetch = fetch; Map = map; } public override Haxl ToHaxlFetch(string bindTo, Scope parentScope) { return Fetch.ToHaxlFetch(bindTo, parentScope).Map(scope => { var newScope = scope; if (scope.InScope(bindTo)) { var value = scope.GetValue(bindTo); newScope = new SelectScope(value, scope); } var blockNumber = newScope.GetLatestBlockNumber(); var rebinder = new RebindToScope() { BlockCount = blockNumber }; var rewritten = rebinder.Rebind(Map); return scope.Add(bindTo, rewritten.Compile().DynamicInvoke(newScope)); }); } } /// /// Monad instance for Fetch. /// public static class ExprExt { public static Fetch Select(this Fetch self, Expression> f) { var isLet = LetExpression.IsLetExpression(f); if (!isLet) return new Select(self, f); Expression>> letBind = _ => self; var letProject = LetExpression.RewriteLetExpression(f); var letPair = new BindProjectPair(letBind, letProject); return new Bind(self.CollectedExpressions.Append(letPair), self) {IsLet = true}; } public static Fetch SelectMany(this Fetch self, Expression>> bind, Expression> project) { var bindExpression = new BindProjectPair(bind, project); var newBinds = self.CollectedExpressions.Append(bindExpression); return new Bind(newBinds, self); } public static Fetch> SelectFetch(this IEnumerable list, Func> bind) { return new RequestSequence(list, bind); } public static async Task FetchWith(this Fetch fetch, Fetcher fetcher, HaxlCache cache, Action logger) { var run = fetch.ToHaxlFetch(HAXL_RESULT_NAME, Scope.New()); var scope = await RunFetch.Run(run, Scope.New(), fetcher.FetchBatch, cache, logger); var result = (A)scope.GetValue(HAXL_RESULT_NAME); logger(Info("==== Result ====")); logger(Info($"{result}")); return result; } } } ================================================ FILE: HaxlSharp.Core/Fetcher.cs ================================================ using System.Collections.Generic; using System.Threading.Tasks; namespace HaxlSharp { /// /// Fetches a request. /// public interface Fetcher { Task FetchBatch(IEnumerable requests); Task Fetch(Fetch request); } } ================================================ FILE: HaxlSharp.Core/HaxlLogEntry.cs ================================================ using System; using System.Collections.Generic; using System.Linq; using System.Text; using System.Threading.Tasks; namespace HaxlSharp { public interface HaxlLogEntry { X Match(Func info, Func warn, Func error); string ToDefaultString(); } public abstract class BaseLogEntry : HaxlLogEntry { public readonly DateTime Timestamp; public abstract string Message { get; } public abstract string Type { get; } public BaseLogEntry() { Timestamp = DateTime.Now; } public abstract X Match(Func info, Func warn, Func error); public string ToDefaultString() { return $"[{Timestamp}] {Type.PadLeft(5)}: {Message}"; } } public class InformationLogEntry : BaseLogEntry { public override string Message { get; } public override string Type => "INFO"; public InformationLogEntry(string info) { Message = info; } public override X Match(Func info, Func warn, Func error) { return info(this); } } public class WarningLogEntry : BaseLogEntry { public override string Message { get; } public override string Type => "WARN"; public WarningLogEntry(string warning) { Message = warning; } public override X Match(Func info, Func warn, Func error) { return warn(this); } } public class ErrorLogEntry : BaseLogEntry { public override string Message { get; } public override string Type => "ERROR"; public ErrorLogEntry(string error) { Message = error; } public override X Match(Func info, Func warn, Func error) { return error(this); } } } ================================================ FILE: HaxlSharp.Core/HaxlSharp.Core.csproj ================================================  10.0 Debug AnyCPU {56487EB5-C699-4EAA-B384-C6F9E64635C5} Library Properties HaxlSharp HaxlSharp.Core v4.5 Profile7 512 {786C830F-07A1-408B-BD7F-6EE04809D6DB};{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC} true full false bin\Debug\ DEBUG;TRACE prompt 4 pdbonly true bin\Release\ TRACE prompt 4 ================================================ FILE: HaxlSharp.Core/HaxlSharp.Core.nuspec ================================================ $id$ $version$ $title$ joashc https://github.com/joashc/HaxlSharp false Composable data fetching with automatic concurrency and request deduplication. Contains only the core HaxlSharp functionality. If you don't intend to implement your own fetcher, install the "HaxlSharp" package instead. Initial release. Copyright © 2016 Joash Chong ================================================ FILE: HaxlSharp.Core/Internal/Applicative/HaxlApplicative.cs ================================================ using System; using System.Collections.Generic; using System.Linq; using static HaxlSharp.Internal.Base; namespace HaxlSharp.Internal { public static class HaxlApplicative { /// /// Converts a project expression to Haxl monad. /// public static Func ProjectToHaxl(ProjectStatement project, string parentBind) { return scope => Haxl.FromFunc((cache, logger) => { var rewritten = RebindToScope.Rebind(project.Expression); var result = rewritten.Compile().DynamicInvoke(scope); return Done.New(_ => { if (project.Expression.BindVariable == HAXL_RESULT_NAME && !scope.IsRoot && parentBind != null) { return scope.WriteParent(parentBind, result); } return scope.Add(project.Expression.BindVariable, result); }); }); } /// /// Converts a single bind expression to the Haxl monad. /// /// /// public static Func BindToHaxl(BindStatement bind) { return scope => { var rewritten = RebindToScope.Rebind(bind.Expression); var value = rewritten.Compile().DynamicInvoke(scope); var wrapped = (Fetchable)value; return wrapped.ToHaxlFetch(bind.Expression.BindVariable, scope); }; } /// /// Converts to Haxl monad, dispatching on statement type. /// public static Func StatementToHaxl(Statement statement, string parentBind) { return statement.Match( BindToHaxl, project => ProjectToHaxl(project, parentBind)); } /// /// Folds an applicative group into a Haxl monad. /// public static Func ApplicativeToHaxl(ApplicativeGroup applicative, string parentBind) { var expressions = applicative.Expressions; if (applicative.Expressions.Count == 1) return StatementToHaxl(expressions.First(), parentBind); return scope => applicative.Expressions.Aggregate ( Haxl.FromFunc((c, l) => Done.New(s => s)), (group, be) => { var haxl = StatementToHaxl(be, parentBind)(scope); return group.Applicative(haxl); } ); } /// /// Converts a list of applicative groups into a Haxl monad. /// public static Haxl ToFetch(List split, string parentBind, Scope parentScope) { if (parentScope == null) parentScope = Scope.New(); Haxl finalFetch = null; Action> bindToFinal = f => { finalFetch = finalFetch == null ? f(parentScope) : finalFetch.Bind(f); }; foreach (var applicative in split) { bindToFinal(ApplicativeToHaxl(applicative, parentBind)); } return finalFetch; } } } ================================================ FILE: HaxlSharp.Core/Internal/Applicative/SplitApplicative.cs ================================================ using System; using System.Collections.Generic; using System.Linq; using System.Linq.Expressions; using static HaxlSharp.Internal.Base; using System.Text.RegularExpressions; namespace HaxlSharp.Internal { public class SplitApplicative { /// /// Splits a monadic bind into applicative groups. /// public static List SplitBind(IEnumerable collectedExpressions, LambdaExpression initial) { var vars = collectedExpressions.Select(GetVariables).ToList(); var numbered = NumberBlocks(vars).ToList(); return MakeApplicative(initial, numbered); } /// /// Gets the variables in a (bind, project) pair. /// private static QueryStatement GetVariables(BindProjectPair pair) { var project = pair.Project; var isLet = project.Parameters.Any(param => param.Name.StartsWith(LET_PREFIX)); if (isLet) { var letParam = project.Parameters.ElementAt(1); var letName = letParam.Name; var originalName = Regex.Split(letName, LET_PREFIX)[1]; var letExpression = Expression.Lambda(project.Body, project.Parameters.First(), Expression.Parameter(letParam.Type, originalName)); var letVariables = ParseExpression.GetExpressionVariables(letExpression); return new LetStatement(originalName, letExpression, letVariables); } var bindVars = ParseExpression.GetExpressionVariables(pair.Bind); var projectVars = ParseExpression.GetExpressionVariables(project); return new BindProjectStatement(pair, bindVars, projectVars) { IsSelect = pair.IsSelect }; } /// /// Appends numbers indicating the block a statement belongs to. /// /// A statement written in the format: /// /// > var nested = from x in a /// > from y in b /// > select x + y; /// > /// > var fetch = from x in nested /// > from y in b /// > select x + y; /// /// will be rewritten as: /// /// > a.SelectMany( /// > x => b, /// > x, y => x + y /// > ).SelectMany( /// > x => b /// > x, y => x + y /// > ); /// /// Because there are two variables named "x", and these /// expressions are written inline, we need to number them, /// in case they end up in the same applicative group. /// /// public static IEnumerable NumberBlocks(List statements) { var blockNumber = 0; var statementCounter = 0; var numStatements = statements.Count(); var isFirst = true; foreach (var statement in statements) { statementCounter++; // SplitBind functions that take one non-transparent parameter are binding the entire result from another monad. // This will hide any variables that were in scope in that monad. statement.StartsBlock = isFirst; if (statement.Match( bind => bind.BindVariables.ParameterNames.Count == 1 && !ParseExpression.IsTransparent(bind.Expressions.Bind.Parameters.First()), let => false) ) { blockNumber++; statement.StartsBlock = true; } statement.BlockNumber = blockNumber; if (statementCounter == numStatements) statement.IsFinal = true; isFirst = false; yield return statement; } } /// /// Groups statements that can be fetched concurrently. /// public static List MakeApplicative(LambdaExpression initial, IEnumerable statements) { var applicatives = new List(); LambdaExpression previousProject = initial; ExpressionVariables previousProjectVars = null; var currentApplicative = new List(); var boundInGroup = new List(); Action split = () => { if (currentApplicative.Any()) applicatives.Add(new ApplicativeGroup(currentApplicative)); currentApplicative = new List(); boundInGroup.Clear(); }; var first = true; foreach (var statement in statements) { var blockNumber = statement.BlockNumber; Func boundExpression = (e, s) => new BoundExpression(e, s, blockNumber); statement.Match( bind => { // The result of the previous monad is bound to this variable name. var previousBindName = bind.BindVariables.ParameterNames.First(); var prefixed = PrefixedVariable(blockNumber, previousBindName); if (first) // Add the initial fetch. { currentApplicative.Add(new BindStatement(boundExpression(initial, prefixed))); } var shouldSplit = ShouldSplit(bind.BindVariables, boundInGroup); if (shouldSplit) split(); // If we're at the beginning of a new block, we should add the previous project statement. if (bind.StartsBlock && !first) { var splitBlock = previousProjectVars != null && ShouldSplit(previousProjectVars, boundInGroup); if (splitBlock) split(); boundInGroup.Clear(); currentApplicative.Add( // This project was from the previous block, so we subtract one here. new ProjectStatement(new BoundExpression(previousProject, prefixed, blockNumber - 1))); if (shouldSplit) split(); } // The result of the current monad is bound to the second parameter of the project fuction: // x.SelectMany( // a => m a, // a, b => new { a, b } // // ^ this b is the result of m a. // ) var bindName = bind.ProjectVariables.ParameterNames.Last(); var prefixedBindName = PrefixedVariable(blockNumber, bindName); currentApplicative.Add(new BindStatement(boundExpression(bind.Expressions.Bind, prefixedBindName))); // We take the final projection function and bind it to the HAXL_RESULT_NAME constant. if (bind.IsFinal) { split(); currentApplicative.Add(new ProjectStatement(boundExpression(bind.Expressions.Project, HAXL_RESULT_NAME))); } // Push out the project function and its variables in case it's the final select of // a nested block and we need to bind it. previousProject = bind.Expressions.Project; previousProjectVars = bind.ProjectVariables; // If we've split the only dependency is the current monad. if (shouldSplit) boundInGroup.Add(bindName); else boundInGroup.AddRange(bind.ProjectVariables.ParameterNames); return UnitVal; }, let => { var paramNames = let.Variables.ParameterNames; var previousBindName = paramNames.First(); var prefixed = PrefixedVariable(blockNumber, previousBindName); if (ShouldSplit(previousProjectVars, boundInGroup)) split(); // The initial lambda is always returns a monad, so we place it into a bind. if (first) currentApplicative.Add(new BindStatement(boundExpression(previousProject, prefixed))); else if (!LetExpression.IsLetExpression(previousProject)) currentApplicative.Add(new ProjectStatement(boundExpression(previousProject, prefixed))); boundInGroup.Add(let.Variables.ParameterNames.First()); if (ShouldSplit(let.Variables, boundInGroup)) split(); boundInGroup.Add(let.Name); currentApplicative.Add(new ProjectStatement(boundExpression(let.Expression, PrefixedVariable(blockNumber, let.Name)))); return UnitVal; } ); first = false; } if (currentApplicative.Any()) applicatives.Add(new ApplicativeGroup(currentApplicative)); return applicatives; } /// /// Checks if this expression binds any variables bound in the current group. /// private static bool ShouldSplit(ExpressionVariables vars, List boundInGroup) { if (vars == null) return false; if (vars.BindsNonTransparentParam) return true; if (vars.Bound.Any(boundInGroup.Contains)) return true; return false; } } } ================================================ FILE: HaxlSharp.Core/Internal/Base/Base.cs ================================================ using System; using System.Collections.Generic; using System.Linq; using System.Text; using System.Threading.Tasks; namespace HaxlSharp.Internal { /// /// Contains constructor functions. /// public static partial class Base { public static IEnumerable Append(this IEnumerable list, A value) { var appendList = new List(list); appendList.Add(value); return appendList; } public static ShowList ShowList(IEnumerable list) { return new ShowList(list); } public static InformationLogEntry Info(string info) { return new InformationLogEntry(info); } public static WarningLogEntry Warn(string warn) { return new WarningLogEntry(warn); } public static ErrorLogEntry Error(string error) { return new ErrorLogEntry(error); } public static Unit UnitVal = new Unit(); } } ================================================ FILE: HaxlSharp.Core/Internal/Base/ByteString.cs ================================================ using System; using System.Collections.Generic; using System.Linq; using System.Text; using System.Threading.Tasks; namespace HaxlSharp.Internal { public static partial class Base { public static readonly Func StringBytes = new UTF8Encoding().GetBytes; public static readonly Func ToLowerHexString = bs => bs.Aggregate(new StringBuilder(32), (sb, b) => sb.Append(b.ToString("x2"))).ToString(); } } ================================================ FILE: HaxlSharp.Core/Internal/Base/Func.cs ================================================ using System; using System.Collections.Generic; using System.Linq; using System.Text; using System.Threading.Tasks; namespace HaxlSharp.Internal { public static partial class Base { public static Func func(Func func) { return func; } public static Func func(Func func) { return func; } public static Func func(Func func) { return func; } public static Func func(Func func) { return func; } public static Func compose(Func f2, Func f1) { return x => f2(f1(x)); } public static Func compose(Func f3, Func f2, Func f1) { return x => f3(f2(f1(x))); } public static Func compose(Func f4, Func f3, Func f2, Func f1) { return x => f4(f3(f2(f1(x)))); } public static Func compose(Func f5, Func f4, Func f3, Func f2, Func f1) { return x => f5(f4(f3(f2(f1(x))))); } } } ================================================ FILE: HaxlSharp.Core/Internal/Base/HaxlConstants.cs ================================================  using System; using System.Text.RegularExpressions; namespace HaxlSharp.Internal { public static partial class Base { /// /// We prefix with "<>" so we can't clash with actual bound variable names. /// public const string HAXL_RESULT_NAME = "<>HAXL_RESULT"; /// /// All transparent identifiers start with this prefix. /// public const string TRANSPARENT_PREFIX = "<>h__Trans"; /// /// We mark let expressions with this prefix. /// public const string LET_PREFIX = "<>HAXL_LET"; /// /// Annotate let arguments with the let prefix. /// public static string PrefixLet(string letVarName) { return $"{LET_PREFIX}{letVarName}"; } /// /// Combines variable names with block numbers. /// public static string PrefixedVariable(int blockNumber, string variableName) { return $"({blockNumber}) {variableName}"; } public static int GetBlockNumber(string bindTo) { if (bindTo == HAXL_RESULT_NAME) return 0; var regex = @"^\((\d+)\).*$"; var match = Regex.Match(bindTo, regex); if (match.Groups.Count < 2) throw new ArgumentException("Invalid bind variable name"); return int.Parse(match.Groups[1].Value); } } } ================================================ FILE: HaxlSharp.Core/Internal/Expressions/LetExpression.cs ================================================ using System; using System.Linq; using System.Linq.Expressions; using static HaxlSharp.Internal.Base; namespace HaxlSharp.Internal { public static class LetExpression { /// /// Checks if a select expression is a Let. /// /// /// This is not a reliable test; it only checks if: /// - The lambda body is just a new expression /// - It returns an anonymous type /// - The parameter name is the same as the first member /// /// If we were to write a select with all these attributes: /// /// public static bool IsLetExpression(LambdaExpression expression) { if (expression.Body.NodeType != ExpressionType.New) return false; var newExpression = (NewExpression)expression.Body; if (newExpression.Arguments.Count != 2) return false; if (!expression.ReturnType.Name.StartsWith("<>f__AnonymousType")) return false; var paramName = expression.Parameters.First().Name; if (paramName != newExpression.Members.First().Name) return false; return true; } public static LambdaExpression RewriteLetExpression(Expression> expression) { var body = (NewExpression)expression.Body; var letVar = body.Arguments.ElementAt(1); var letParam = body.Members.ElementAt(1); return Expression.Lambda(letVar, expression.Parameters.First(), Expression.Parameter(letVar.Type, PrefixLet(letParam.Name))); } } } ================================================ FILE: HaxlSharp.Core/Internal/Expressions/ParameterAccessVisitor.cs ================================================ using System.Collections.Generic; using System.Linq.Expressions; namespace HaxlSharp.Internal { /// /// Recursively collects the parameter and member accesses of a given expression. /// public class ParameterAccessVisitor : ExpressionVisitor { public readonly List ParameterAccesses; public readonly List MemberAccesses; public ParameterAccessVisitor() { ParameterAccesses = new List(); MemberAccesses = new List(); } protected override Expression VisitParameter(ParameterExpression node) { ParameterAccesses.Add(node); return base.VisitParameter(node); } protected override Expression VisitMember(MemberExpression node) { MemberAccesses.Add(node); return base.VisitMember(node); } } } ================================================ FILE: HaxlSharp.Core/Internal/Expressions/ParseExpression.cs ================================================ using System; using System.Collections.Generic; using System.Linq; using System.Linq.Expressions; using System.Reflection; using static HaxlSharp.Internal.Base; namespace HaxlSharp.Internal { public static class ParseExpression { /// /// Gets the free and bound variables of a lambda expression. /// public static ExpressionVariables GetExpressionVariables(LambdaExpression bind) { var visitor = new ParameterAccessVisitor(); visitor.Visit(bind.Body); var bound = visitor.MemberAccesses .Select(MemberAccess) .Where(m => !m.Name.StartsWith(TRANSPARENT_PREFIX) && m.FromTransparent) .Select(f => f.Name) .ToList(); var paramVisitor = new ParameterAccessVisitor(); foreach (var param in bind.Parameters) { paramVisitor.Visit(param); } var bindsNonTransparent = bind.Parameters.Any( bindParam => !IsTransparent(bindParam) && BindsNonTransparentParam(visitor.ParameterAccesses, bindParam.Name)); return new ExpressionVariables(bindsNonTransparent, bound, paramVisitor.ParameterAccesses.SelectMany(MemberNames).Select(f => f.Name).ToList()); } /// /// Get all member names within a parameter expression. /// private static IEnumerable MemberNames(ParameterExpression parameter) { // Transparent identifiers start with this prefix // If we have a transparent identifier, we pull out the appropriate members. if (parameter.Name.StartsWith(TRANSPARENT_PREFIX)) { var properties = parameter.Type.GetRuntimeProperties(); return from property in properties where !property.Name.StartsWith(TRANSPARENT_PREFIX) select new FreeVariable(property.Name, true); } return new List { new FreeVariable(parameter.Name, false) }; } private static bool BindsNonTransparentParam(List parameterExpressions, string paramName) { return parameterExpressions.Any(pe => pe.Name == paramName); } /// /// Checks if a given member expression ultimately points to a transparent identifier. /// public static bool IsFromTransparent(MemberExpression expression) { if (expression.Expression == null) return false; switch (expression.Expression.NodeType) { case ExpressionType.Parameter: return IsTransparent((ParameterExpression)expression.Expression); case ExpressionType.Constant: return false; } return IsFromTransparent(expression.Expression as MemberExpression); } public static bool IsTransparent(ParameterExpression expression) { return expression.Name.StartsWith(TRANSPARENT_PREFIX); } /// /// /// public static bool IsTransparentMember(MemberExpression expression) { return expression.Member.Name.StartsWith(TRANSPARENT_PREFIX); } /// /// /// private static FreeVariable MemberAccess(MemberExpression argument) { //return new FreeVariable(argument.Member.Name, false); var fromTransparent = IsFromTransparent(argument); if (!fromTransparent) return new FreeVariable(argument.Member.Name, false); switch (argument.Expression.NodeType) { case ExpressionType.Parameter: return new FreeVariable(argument.Member.Name, true); case ExpressionType.MemberAccess: { var member = (MemberExpression) argument.Expression; return member.Member.Name.StartsWith(TRANSPARENT_PREFIX) ? new FreeVariable(argument.Member.Name, true) : MemberAccess(member); } } throw new ArgumentException("Error getting transparent identifier name"); } } } ================================================ FILE: HaxlSharp.Core/Internal/Expressions/RebindToScope.cs ================================================ using System; using System.Collections.Generic; using System.Linq; using System.Linq.Expressions; using System.Reflection; using System.Text; using System.Threading.Tasks; using static HaxlSharp.Internal.Base; namespace HaxlSharp.Internal { /// /// Replaces all transparent identifier or parameter accessor expressions with expressions that read from scope parameter. /// public class RebindToScope : ExpressionVisitor { private List paramNames; private ParameterExpression scopeParam; public int BlockCount { get; set; } private MethodInfo GetValue = typeof(Scope).GetRuntimeMethod("GetValue", new Type[] { typeof(string) }); /// /// Replace all parameters with a single scope parameter, then rewrite body to read from that scope. /// public LambdaExpression Rebind(LambdaExpression lambda) { paramNames = lambda.Parameters.Select(p => p.Name).Where(n => !n.StartsWith(TRANSPARENT_PREFIX)).ToList(); scopeParam = Expression.Parameter(typeof(Scope), "scope"); var newExpression = Expression.Lambda( lambda.Body, new ParameterExpression[] { scopeParam } ); return (LambdaExpression)base.Visit(newExpression); } public static LambdaExpression Rebind(BoundExpression expression) { var rebinder = new RebindToScope {BlockCount = expression.BlockNumber}; return rebinder.Rebind(expression.Expression); } /// /// We only want to rewrite transparent identifier accessors. /// protected override Expression VisitMember(MemberExpression node) { if (ParseExpression.IsFromTransparent(node) && !ParseExpression.IsTransparentMember(node)) { return RewritePropertyAccess(node); } return base.VisitMember(node); } /// /// Rewrites transparent identifier accessors. /// /// /// If we have a nested accessor, like: /// > ti0.ti1.a.b.c /// this will rewrite the expression as: /// > SCOPE.a.b.c /// private Expression RewritePropertyAccess(MemberExpression node) { Expression current = node; var expressionStack = new Stack(); while (current.NodeType == ExpressionType.MemberAccess) { var memberAccess = ((MemberExpression)current); if (!memberAccess.Member.Name.StartsWith(TRANSPARENT_PREFIX)) expressionStack.Push(current); current = memberAccess.Expression; } var property = (MemberExpression)expressionStack.Pop(); var propertyName = property.Member.Name; var value = Expression.Call(scopeParam, GetValue, Expression.Constant(PrefixedVariable(BlockCount, propertyName))); Expression rewritten = Expression.Convert(value, property.Type); while (expressionStack.Any()) { var top = expressionStack.Pop(); rewritten = Expression.MakeMemberAccess(rewritten, ((MemberExpression)top).Member); } return rewritten; } /// /// Rewrites parameter accessors to read from scope. /// protected override Expression VisitParameter(ParameterExpression node) { if (paramNames.Contains(node.Name)) { var memberType = node.Type; var memberName = node.Name; var value = Expression.Call(scopeParam, GetValue, Expression.Constant(PrefixedVariable(BlockCount, memberName))); return Expression.Convert(value, memberType); } return node; } } } ================================================ FILE: HaxlSharp.Core/Internal/Haxl.cs ================================================ using System; using System.Linq; using static HaxlSharp.Internal.Base; namespace HaxlSharp.Internal { /// /// The Haxl monad. /// public class Haxl { /// /// The result of the fetch. /// /// /// Accessing a Done result before it's fetched is a framework error. /// public readonly Func, Result> Result; public Haxl(Func, Result> result) { Result = result; } public static Haxl FromFunc(Func, Result> resultFunc) { return new Haxl(resultFunc); } /// /// Applicative pure. /// /// We need a bind variable string because our applicative instance is specialized to (Scope -> Scope) functions. /// /// public static Haxl Pure(string bindTo, object value) { return FromFunc((cache, logger) => Done.New(scope => scope.Add(bindTo, value))); } /// /// Identity >>= (scope -> x) = x = x >>= (scope -> Identity) /// public static Haxl Identity() { return FromFunc((cache, logger) => Done.New(s => s)); } public Haxl Map(Func addResult) { return new Haxl((cache, logger) => { var result = Result(cache, logger); return result.Match( done => Done.New(compose(addResult, done.AddToScope)), blocked => Blocked.New(blocked.BlockedRequests, blocked.Continue.Map(addResult)) ); }); } } public static class HaxlFetchExt { /// /// Monad instance for HaxlFetch. /// public static Haxl Bind(this Haxl fetch, Func bind) { return Haxl.FromFunc((cache, logger) => { var result = fetch.Result(cache, logger); return result.Match( done => bind(done.AddToScope(Scope.New())).Result(cache, logger), blocked => Blocked.New(blocked.BlockedRequests, blocked.Continue.Bind(bind)) ); }); } /// /// "Applicative" instance for HaxlFetch. /// /// /// This isn't a true applicative instance; we don't have: /// /// > (<*>) :: f (a -> b) -> f a -> f b /// /// In Haskell Haxl, the applicative instance is used to keep fetched values in scope: /// /// > (a, b) <- (,) <$> fetch1 <*> fetch2 /// /// C# can't do nested lambda scoping, and uses transparent identifers instead. /// Because the transparent identifers aren't accessible to us, we use our own scoping system. /// /// This means our (a -> b) function is *always* (Scope -> Scope); /// we therefore can write our "Applicative" instance as simply a function that takes two Fetches. /// public static Haxl Applicative(this Haxl fetch1, Haxl fetch2) { return Haxl.FromFunc((cache, logger) => { var result1 = fetch1.Result(cache, logger); var result2 = fetch2.Result(cache, logger); return result1.Match ( done1 => result2.Match ( done2 => Done.New(compose(done2.AddToScope, done1.AddToScope)), blocked2 => Blocked.New(blocked2.BlockedRequests, blocked2.Continue.Map(done1.AddToScope)) ), blocked1 => result2.Match ( done2 => Blocked.New(blocked1.BlockedRequests, blocked1.Continue.Map(done2.AddToScope)), blocked2 => Blocked.New( blocked1.BlockedRequests.Concat(blocked2.BlockedRequests), blocked1.Continue.Applicative(blocked2.Continue) ) ) ); }); } } } ================================================ FILE: HaxlSharp.Core/Internal/HaxlCache.cs ================================================ using System; using System.Collections.Generic; namespace HaxlSharp.Internal { /// /// Caches results per-request for deduplication. /// public class HaxlCache { private readonly CacheKeyGenerator _keyGenerator; private readonly Dictionary _cache; public HaxlCache(CacheKeyGenerator generator) { _keyGenerator = generator; _cache = new Dictionary(); } public CacheResult Lookup(Returns request) { var key = _keyGenerator.ForRequest(request); if (!_cache.ContainsKey(key)) return CacheResult.NotFound; var blockedRequest = _cache[key]; return CacheResult.Found(blockedRequest); } public void Insert(Returns request, BlockedRequest blocked) { var key = _keyGenerator.ForRequest(request); if (_cache.ContainsKey(key)) throw new Exception("Internal Haxl error: attempted to cache duplicate request."); _cache[key] = blocked; } } } ================================================ FILE: HaxlSharp.Core/Internal/Result.cs ================================================ using System; using System.Collections.Generic; namespace HaxlSharp.Internal { /// /// The result of a fetch. /// public interface Result { X Match(Func done, Func blocked); } /// /// The result of a completed fetch. /// public class Done : Result { public Func AddToScope; public static Done New(Func addToScope) { return new Done(addToScope); } public Done(Func addToScope) { AddToScope = addToScope; } public X Match(Func done, Func blocked) { return done(this); } } /// /// A result that's blocked on one or more requests. /// public class Blocked : Result { public readonly IEnumerable BlockedRequests; public readonly Haxl Continue; public static Blocked New(IEnumerable blocked, Haxl cont) { return new Blocked(blocked, cont); } private Blocked(IEnumerable blocked, Haxl cont) { BlockedRequests = blocked; Continue = cont; } public X Match(Func done, Func blocked) { return blocked(this); } } } ================================================ FILE: HaxlSharp.Core/Internal/RunFetch.cs ================================================ using System; using System.Collections.Generic; using System.Threading.Tasks; namespace HaxlSharp.Internal { public static class RunFetch { /// /// Repeatedly fetches requests until we have the result. /// public static async Task Run(Haxl fetch, Scope scope, Func, Task> fetcher, HaxlCache cache, Action logger) { var result = fetch.Result(cache, logger); return await result.Match( done => Task.FromResult(done.AddToScope(scope)), async blocked => { await fetcher(blocked.BlockedRequests); return await Run(blocked.Continue, scope, fetcher, cache, logger); } ); } } } ================================================ FILE: HaxlSharp.Core/Internal/Scope.cs ================================================ using System; using System.Collections.Generic; using System.Linq; using static HaxlSharp.Internal.Base; namespace HaxlSharp.Internal { /// /// Scope with simple inheritance that "shadows" variable names. /// public class Scope { private readonly Dictionary boundVariables; private readonly Scope parentScope; public static Scope New() { return new Scope(); } public Scope() { boundVariables = new Dictionary(); parentScope = null; } public Scope(Scope scope) { boundVariables = new Dictionary(); parentScope = scope; } public bool IsRoot => parentScope == null; public virtual object GetValue(string variableName) { if (boundVariables.ContainsKey(variableName)) return boundVariables[variableName]; if (parentScope == null) throw new ArgumentException($"No variable named '{variableName}' in scope."); return parentScope.GetValue(variableName); } public bool InScope(string variableName) { if (boundVariables.ContainsKey(variableName)) return true; if (parentScope == null) return false; return parentScope.InScope(variableName); } public Scope Add(string name, object value) { boundVariables[name] = value; return this; } public int GetLatestBlockNumber() { if (!Keys.Any()) return 0; return Keys.Select(GetBlockNumber).Max(); } public Scope WriteParent(string name, object value) { if (parentScope == null) return Add(name, value); return parentScope.Add(name, value); } public IEnumerable Keys { get { if (parentScope == null) return boundVariables.Keys; return boundVariables.Keys.Concat(parentScope.Keys); } } public IEnumerable ShallowValues => boundVariables.Values; } /// /// A specialized scope object that will return a fixed value for the first unknown key request. /// public class SelectScope : Scope { private readonly object _selectValue; private string _unknownName; public SelectScope(object selectValue, Scope scope) : base(scope) { _selectValue = selectValue; } public override object GetValue(string variableName) { try { return base.GetValue(variableName); } catch (ArgumentException) { if (_unknownName != null && variableName != _unknownName) throw new ArgumentException("Attempted to access more than one unknown variable."); _unknownName = variableName; return _selectValue; } } } } ================================================ FILE: HaxlSharp.Core/Internal/Types/ApplicativeGroup.cs ================================================ using System.Collections.Generic; namespace HaxlSharp.Internal { public class ApplicativeGroup { public readonly List Expressions; public ApplicativeGroup(List expressions) { Expressions = expressions; } } } ================================================ FILE: HaxlSharp.Core/Internal/Types/BindProjectPair.cs ================================================ using System.Linq.Expressions; namespace HaxlSharp.Internal { /// /// A pair of (bind, project) expressions that form part of a query expression. /// /// /// Given the following query: /// /// > from x in a // (1) /// > from y in b(x) // (2) /// > from z in c(x) // (3) /// > select (x, y, z) // (4) /// /// It will be desugared into: /// /// a.SelectMany( // Line (1) Initial /// x => b, // Line (2) SplitBind /// (x, y) => new { x, y } // Line (2) Project /// ) /// .SelectMany( /// ti0 => c(ti0.x), // Line (3) SplitBind /// (ti0, z) => (ti0.x, ti0.y, z) // Line (4) Final Project /// ); /// /// We can represent any query expression as an initial expression, along with a list of (bind, project) pairs. /// public class BindProjectPair { public BindProjectPair(LambdaExpression bind, LambdaExpression project) { Bind = bind; Project = project; } public bool IsSelect { get; set; } public readonly LambdaExpression Bind; public readonly LambdaExpression Project; } } ================================================ FILE: HaxlSharp.Core/Internal/Types/BoundExpression.cs ================================================ using System.Linq.Expressions; namespace HaxlSharp.Internal { /// /// An expression that is bound to a particular variable name. /// public class BoundExpression { public readonly LambdaExpression Expression; public readonly string BindVariable; public readonly int BlockNumber; public BoundExpression(LambdaExpression expression, string bindVariable, int blockNumber) { Expression = expression; BindVariable = bindVariable; BlockNumber = blockNumber; } } } ================================================ FILE: HaxlSharp.Core/Internal/Types/CacheResult.cs ================================================ using System; using static HaxlSharp.Internal.Base; namespace HaxlSharp.Internal { /// /// The result of a cache lookup. /// public abstract class CacheResult { public abstract X Match(Func notFound, Func found); public static NotFound NotFound = new NotFound(); public static Found Found(BlockedRequest request) { return new Found(request); } } /// /// An item matching the cache key was not found. /// public class NotFound : CacheResult { public override X Match(Func notFound, Func found) { return notFound(UnitVal); } } /// /// The item matching the cache key. /// public class Found : CacheResult { private readonly BlockedRequest _blocked; public Found(BlockedRequest blocked) { _blocked = blocked; } public override X Match(Func notFound, Func found) { return found(_blocked); } } } ================================================ FILE: HaxlSharp.Core/Internal/Types/ExpressionVariables.cs ================================================ using System.Collections.Generic; namespace HaxlSharp.Internal { /// /// The variables in an expression. /// public class ExpressionVariables { public readonly bool BindsNonTransparentParam; public readonly List Bound; public readonly List ParameterNames; public ExpressionVariables(bool bindsNonTransparentParam, List bound, List parameterNames) { BindsNonTransparentParam = bindsNonTransparentParam; Bound = bound; ParameterNames = parameterNames; } } } ================================================ FILE: HaxlSharp.Core/Internal/Types/FreeVariable.cs ================================================  namespace HaxlSharp.Internal { /// /// A free variable that might come from a transparent identifier. /// public class FreeVariable { public readonly bool FromTransparent; public readonly string Name; public FreeVariable(string name, bool fromTransparent) { Name = name; FromTransparent = fromTransparent; } } } ================================================ FILE: HaxlSharp.Core/Internal/Types/QueryStatement.cs ================================================ using System; using System.Linq.Expressions; namespace HaxlSharp.Internal { /// /// Represents a line in a query expression. /// public interface QueryStatement { X Match(Func bind, Func let); int BlockNumber { get; set; } bool StartsBlock { get; set; } bool IsFinal { get; set; } } /// /// The variables of each expression in a (bind, project) expression pair. /// public class BindProjectStatement : QueryStatement { public readonly BindProjectPair Expressions; public readonly ExpressionVariables BindVariables; public readonly ExpressionVariables ProjectVariables; public bool IsSelect { get; set; } public BindProjectStatement(BindProjectPair expressions, ExpressionVariables bindVars, ExpressionVariables projectVars) { Expressions = expressions; BindVariables = bindVars; ProjectVariables = projectVars; } public X Match(Func bind, Func let) { return bind(this); } public int BlockNumber { get; set; } public bool StartsBlock { get; set; } public bool IsFinal { get; set; } } /// /// A let statement, e.g. /// > from x in a /// > let y = 2 /// > select x + y; /// public class LetStatement : QueryStatement { public readonly LambdaExpression Expression; public readonly ExpressionVariables Variables; public readonly string Name; public LetStatement(string name, LambdaExpression expression, ExpressionVariables variables) { Name = name; Expression = expression; Variables = variables; } public X Match(Func bind, Func let) { return let(this); } public int BlockNumber { get; set; } public bool StartsBlock { get; set; } public bool IsFinal { get; set; } } } ================================================ FILE: HaxlSharp.Core/Internal/Types/ShowList.cs ================================================ using System.Collections; using System.Collections.Generic; using System.Linq; using System.Text; namespace HaxlSharp.Internal { /// /// Simple pretty printing wrapper around IEnumerable. /// public class ShowList : IEnumerable { public readonly IEnumerable List; public ShowList(IEnumerable list) { List = list; } public IEnumerator GetEnumerator() { return List.GetEnumerator(); } public override string ToString() { if (!List.Any()) return "[]"; var builder = new StringBuilder(); builder.Append("[ "); var first = List.First(); var rest = List.Skip(1); builder.Append(first); foreach (var item in rest) { builder.Append($", {item}"); } builder.Append(" ]"); return builder.ToString(); } IEnumerator IEnumerable.GetEnumerator() { return List.GetEnumerator(); } } } ================================================ FILE: HaxlSharp.Core/Internal/Types/Statement.cs ================================================ using System; namespace HaxlSharp.Internal { /// /// Represents an individual statement expression. /// public interface Statement { X Match( Func bind, Func project ); } /// /// A statement that returns a monad type. /// public class BindStatement : Statement { public readonly BoundExpression Expression; public BindStatement(BoundExpression expression) { Expression = expression; } public X Match(Func bind, Func project) { return bind(this); } } /// /// A statement that returns a non-monadic type. /// public class ProjectStatement : Statement { public readonly BoundExpression Expression; public ProjectStatement(BoundExpression expression) { Expression = expression; } public X Match(Func bind, Func project) { return project(this); } } } ================================================ FILE: HaxlSharp.Core/Internal/Types/Unit.cs ================================================ namespace HaxlSharp.Internal { /// /// The only inhabitant of the Unit type. /// public class Unit { } } ================================================ FILE: HaxlSharp.Core/Properties/AssemblyInfo.cs ================================================ using System.Reflection; using System.Runtime.CompilerServices; using System.Runtime.InteropServices; // General Information about an assembly is controlled through the following // set of attributes. Change these attribute values to modify the information // associated with an assembly. [assembly: AssemblyTitle("HaxlSharp.Core")] [assembly: AssemblyDescription("")] [assembly: AssemblyConfiguration("")] [assembly: AssemblyCompany("")] [assembly: AssemblyProduct("HaxlSharp (Core Only)")] [assembly: AssemblyCopyright("Copyright © 2016")] [assembly: AssemblyTrademark("")] [assembly: AssemblyCulture("")] // Setting ComVisible to false makes the types in this assembly not visible // to COM components. If you need to access a type in this assembly from // COM, set the ComVisible attribute to true on that type. [assembly: ComVisible(false)] // The following GUID is for the ID of the typelib if this project is exposed to COM [assembly: Guid("56487eb5-c699-4eaa-b384-c6f9e64635c5")] // Version information for an assembly consists of the following four values: // // Major Version // Minor Version // Build Number // Revision // // You can specify all the values or you can default the Build and Revision Numbers // by using the '*' as shown below: // [assembly: AssemblyVersion("1.0.*")] [assembly: AssemblyVersion("0.1.*")] ================================================ FILE: HaxlSharp.Core/Response.cs ================================================ using System; namespace HaxlSharp { /// /// The result of a primitive request. /// public class Response { public readonly object Value; public readonly Type ResultType; public Response(object value, Type resultType) { Value = value; ResultType = resultType; } } } ================================================ FILE: HaxlSharp.Core/Returns.cs ================================================ namespace HaxlSharp { /// /// A request object annotated with a return type. /// /// The return type of the request. public interface Returns { } public static class RequestExt { public static Fetch ToFetch(this Returns request) { return new Request(request); } } } ================================================ FILE: HaxlSharp.Fetcher/FetcherBuilder.cs ================================================ using System; using System.Collections.Generic; using System.Threading.Tasks; using HaxlSharp.Internal; namespace HaxlSharp { /// /// Constructs a fetcher from request handlers. /// public class FetcherBuilder { private readonly Dictionary> _fetchFunctions; private readonly Dictionary>> _asyncFetchFunctions; private Action _logWith; public FetcherBuilder() { _fetchFunctions = new Dictionary>(); _asyncFetchFunctions = new Dictionary>>(); } public static FetcherBuilder New() { return new FetcherBuilder(); } /// /// Creates untyped fetch function from typed fetch function. /// private Func CreateFetchFunc(Func fetchFunc) where Req : Returns { var resultType = typeof(Res); var requestType = typeof(Req); Func untypedFetchFunc = request => { if (request.RequestType != requestType) throw new ArgumentException("Invalid request type"); var typedRequest = (Req)request.TypedRequest; var result = fetchFunc(typedRequest); return new Response(result, typeof(Res)); }; return untypedFetchFunc; } /// /// Creates untyped async fetch function from typed async fetch function. /// private Func> CreateAsyncFetchFunc(Func> fetchFunc) where Req : Returns { var resultType = typeof(Res); var requestType = typeof(Req); Func> untypedFetchFunc = async blockedRequest => { if (blockedRequest.RequestType != requestType) throw new ArgumentException($"Request type mismatch: expected '{requestType}', got '{blockedRequest.RequestType}'"); var typedRequest = (Req)blockedRequest.TypedRequest; var result = await fetchFunc(typedRequest); return new Response(result, typeof(Res)); }; return untypedFetchFunc; } /// /// Throws exception if there's already a handler registered for given type. /// private void ThrowIfRegistered(Type requestType) { if (!_fetchFunctions.ContainsKey(requestType) && !_asyncFetchFunctions.ContainsKey(requestType)) return; throw new ArgumentException($"Attempted to register multiple handlers for request type '{requestType}'"); } /// /// Adds a request handler to the fetcher. /// public FetcherBuilder FetchRequest(Func fetchFunction) where Req : Returns { var requestType = typeof(Req); ThrowIfRegistered(requestType); _fetchFunctions.Add(requestType, CreateFetchFunc(fetchFunction)); return this; } /// /// Adds an async request handler to the fetcher. /// public FetcherBuilder FetchRequest(Func> fetchFunction) where Req : Returns { var requestType = typeof(Req); ThrowIfRegistered(requestType); _asyncFetchFunctions.Add(requestType, CreateAsyncFetchFunc(fetchFunction)); return this; } public FetcherBuilder LogWith(Action logWith) { _logWith = logWith; return this; } public HaxlFetcher Create() { return new HaxlFetcher(_fetchFunctions, _asyncFetchFunctions, _logWith); } } } ================================================ FILE: HaxlSharp.Fetcher/HashedRequestKey.cs ================================================ using static HaxlSharp.Internal.Base; using Newtonsoft.Json; using Org.BouncyCastle.Crypto.Digests; using Org.BouncyCastle.Utilities.Encoders; namespace HaxlSharp { public class HashedRequestKey : CacheKeyGenerator { public string ForRequest(Returns request) { return StaticForRequest(request); } private static byte[] Md5Hash(byte[] input) { // Update the input of MD5 var md5 = new MD5Digest(); md5.BlockUpdate(input, 0, input.Length); // Get the output and hash it var output = new byte[md5.GetDigestSize()]; md5.DoFinal(output, 0); return Hex.Encode(output); } public static string StaticForRequest(Returns request) { var json = JsonConvert.SerializeObject(request, request.GetType(), new JsonSerializerSettings {TypeNameHandling = TypeNameHandling.All, TypeNameAssemblyFormat = System.Runtime.Serialization.Formatters.FormatterAssemblyStyle.Full}); return compose(ToLowerHexString, Md5Hash, StringBytes)(json); } } } ================================================ FILE: HaxlSharp.Fetcher/HaxlFetcher.cs ================================================ using System; using static HaxlSharp.Internal.Base; using System.Collections.Generic; using System.Diagnostics; using System.Linq; using HaxlSharp.Internal; using System.Threading.Tasks; namespace HaxlSharp { public class HaxlFetcher : Fetcher { private readonly Dictionary> _fetchFunctions; private readonly Dictionary>> _asyncFetchFunctions; bool isLogging; public HaxlFetcher(Dictionary> fetchFunctions, Dictionary>> asyncFetchFunctions, Action onLogEntry = null) { _fetchFunctions = fetchFunctions; _asyncFetchFunctions = asyncFetchFunctions; isLogging = onLogEntry != null; if (isLogging) OnLogEntry += log => onLogEntry(log); else OnLogEntry += log => { }; } private void ThrowIfUnhandled(BlockedRequest request) { if (!_fetchFunctions.ContainsKey(request.RequestType) && !_asyncFetchFunctions.ContainsKey(request.RequestType)) { RaiseLogEntry(Error($"No handler for request type '{request.RequestType}' found.")); throw new Exception($"No handler for request type '{request.RequestType}' found."); } } public async Task Fetch(BlockedRequest request) { ThrowIfUnhandled(request); if (_fetchFunctions.ContainsKey(request.RequestType)) { var handler = _fetchFunctions[request.RequestType]; return await Task.Factory.StartNew(() => { var result = handler(request); request.Resolver.SetResult(result.Value); RaiseLogEntry(Info($"Fetched '{request.BindName}': {result.Value}")); return result; }); } var asyncHandler = _asyncFetchFunctions[request.RequestType]; var response = await asyncHandler(request); request.Resolver.SetResult(response.Value); RaiseLogEntry(Info($"Fetched '{request.BindName}': {response.Value}")); return response; } public async Task FetchBatch(IEnumerable requests) { RaiseLogEntry(Info("==== Batch ====")); var tasks = requests.Select(Fetch); await Task.WhenAll(tasks); } public delegate void HandleLogEntry(HaxlLogEntry logEntry); public event HandleLogEntry OnLogEntry; private void RaiseLogEntry(HaxlLogEntry logEntry) { if (!isLogging) return; OnLogEntry(logEntry); } public async Task Fetch(Fetch request) { var cache = new HaxlCache(new HashedRequestKey()); return await request.FetchWith(this, cache, RaiseLogEntry); } } } ================================================ FILE: HaxlSharp.Fetcher/HaxlSharp.Fetcher.csproj ================================================  10.0 Debug AnyCPU {4D87351F-EEE6-4361-B889-D8217EB314AA} Library Properties HaxlSharp HaxlSharp v4.5 Profile7 512 {786C830F-07A1-408B-BD7F-6EE04809D6DB};{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC} true full false bin\Debug\ DEBUG;TRACE prompt 4 pdbonly true bin\Release\ TRACE prompt 4 Designer Designer {56487EB5-C699-4EAA-B384-C6F9E64635C5} HaxlSharp.Core ..\packages\BouncyCastle.1.8.1\lib\BouncyCastle.Crypto.dll True ..\packages\Newtonsoft.Json.8.0.3\lib\portable-net40+sl5+wp80+win8+wpa81\Newtonsoft.Json.dll True ================================================ FILE: HaxlSharp.Fetcher/HaxlSharp.Fetcher.nuspec ================================================ $id$ $version$ HaxlSharp joashc https://github.com/joashc/HaxlSharp false Composable data fetching with automatic concurrency and request deduplication. Installs the core HaxlSharp functionality as well as a fetcher implementation. Initial release. Copyright © 2016 Joash Chong ================================================ FILE: HaxlSharp.Fetcher/Properties/AssemblyInfo.cs ================================================ using System.Reflection; using System.Runtime.CompilerServices; using System.Runtime.InteropServices; // General Information about an assembly is controlled through the following // set of attributes. Change these attribute values to modify the information // associated with an assembly. [assembly: AssemblyTitle("HaxlSharp.Fetcher")] [assembly: AssemblyDescription("")] [assembly: AssemblyConfiguration("")] [assembly: AssemblyCompany("")] [assembly: AssemblyProduct("HaxlSharp.Fetcher")] [assembly: AssemblyCopyright("Copyright © 2016")] [assembly: AssemblyTrademark("")] [assembly: AssemblyCulture("")] // Setting ComVisible to false makes the types in this assembly not visible // to COM components. If you need to access a type in this assembly from // COM, set the ComVisible attribute to true on that type. [assembly: ComVisible(false)] // The following GUID is for the ID of the typelib if this project is exposed to COM [assembly: Guid("4d87351f-eee6-4361-b889-d8217eb314aa")] // Version information for an assembly consists of the following four values: // // Major Version // Minor Version // Build Number // Revision // // You can specify all the values or you can default the Build and Revision Numbers // by using the '*' as shown below: // [assembly: AssemblyVersion("1.0.*")] [assembly: AssemblyVersion("0.1.*")] ================================================ FILE: HaxlSharp.Fetcher/packages.config ================================================  ================================================ FILE: HaxlSharp.Test/ApplicativeRewriteTest.cs ================================================ using System; using System.Collections.Generic; using System.Threading.Tasks; using System.Linq; using Microsoft.VisualStudio.TestTools.UnitTesting; using System.Diagnostics; using HaxlSharp.Internal; using static HaxlSharp.Test.Blog; using static HaxlSharp.Internal.Base; namespace HaxlSharp.Test { [TestClass] public class ApplicativeRewriteTest { public HaxlFetcher fetcher = Fetcher(); [TestMethod] public async Task SingleFetch_ShouldHaveOneBatch() { var postIds = FetchAllPostIds(); await fetcher.Fetch(postIds); } [TestMethod] public async Task SelectFetch() { var fetch = FetchAllPostIds(); var postIds = await fetcher.Fetch(fetch); var first = postIds.First(); var fetch1 = from ids in FetchAllPostIds().Select(list => list.Select(x => x + 1)) from somethingElse in FetchAllPostIds() select ids.First(); var firstPlus1 = await fetcher.Fetch(fetch1); Assert.AreEqual(first + 1, firstPlus1); } [TestMethod] public async Task SelectFetchFinal() { var fetch = FetchAllPostIds(); var postIds = await fetcher.Fetch(fetch); var first = postIds.First(); var fetch1 = from ids in FetchAllPostIds().Select(list => list.Select(x => x + 1)) select ids.First(); var firstPlus1 = await fetcher.Fetch(fetch1); Assert.AreEqual(first + 1, firstPlus1); } [TestMethod] public async Task SelectLetFetch() { var fetch = FetchAllPostIds(); var postIds = await fetcher.Fetch(fetch); var first = postIds.First(); var fetch1 = from ids in FetchAllPostIds().Select(list => list.Select(x => x + 1)) let first1 = ids.First() from ids2 in FetchAllPostIds() select first1; var firstPlus1 = await fetcher.Fetch(fetch1); Assert.AreEqual(first + 1, firstPlus1); } [TestMethod] public async Task SelectLetFetchExtended() { var fetch = FetchAllPostIds(); var postIds = await fetcher.Fetch(fetch); var first = postIds.First(); var fetch1 = from ids in FetchAllPostIds().Select(list => list.Select(x => x + 1)) let first1 = ids.First() from ids2 in FetchAllPostIds() from ids3 in FetchAllPostIds() select first1 + ids.First(); var firstPlus1 = await fetcher.Fetch(fetch1); Assert.AreEqual(first + 1 + 1, firstPlus1); } [TestMethod] public async Task SequentialFetch_ShouldHaveTwoBatches() { var firstPostInfo = from postIds in FetchAllPostIds() from firstInfo in FetchPostInfo(postIds.First()) select firstInfo; var result = await fetcher.Fetch(firstPostInfo); } [TestMethod] public async Task SequentialFetch_ShouldHaveTwoBatchesRepeat() { var firstPostInfo = from postIds in FetchAllPostIds() from firstInfo in FetchPostInfo(postIds.Skip(1).First()) select firstInfo; var result = await fetcher.Fetch(firstPostInfo); } [TestMethod] public async Task Sequence_ShouldBeApplicative() { var getAllPostsInfo = from postIds in FetchAllPostIds() from postInfo in postIds.SelectFetch(GetPostDetails) select postInfo; var result = await fetcher.Fetch(getAllPostsInfo); } [TestMethod] public async Task Sequence_ShouldBeApplicativeAgain() { var getAllPostsInfo = from postIds in FetchAllPostIds() from postInfo in postIds.SelectFetch(GetPostDetails) select postInfo; var result = await fetcher.Fetch(getAllPostsInfo); } [TestMethod] public async Task Sequence_ShouldBeApplicativeAgainAddOne() { var getAllPostsInfo = from postIds in FetchAllPostIds() from postInfo in postIds.Select(x => x + 1).SelectFetch(GetPostDetails) select postInfo; var result = await fetcher.Fetch(getAllPostsInfo); } [TestMethod] public async Task SharedDependency() { var fetch = from postIds in FetchAllPostIds() from postInfo in postIds.SelectFetch(Blog.FetchPostInfo) from firstPostInfo in FetchPostInfo(postIds.First()) select firstPostInfo; var result = await fetcher.Fetch(fetch); } [TestMethod] public async Task JustSequence() { var sequence = Enumerable.Range(0, 10).SelectFetch(Blog.FetchPostInfo); var result = await fetcher.Fetch(sequence); } [TestMethod] public async Task GetDuplicateFriends() { var fetch = from ids in FetchTwoLatestPosts() from friends in FetchPostAuthorFriends(ids.Item1) from friends2 in FetchPostAuthorFriends(ids.Item2) select ShowList(friends.Concat(friends2)); var result = await fetcher.Fetch(fetch); ; } [TestMethod] public async Task GetFriends() { var fetch = from info in FetchPostInfo(3) from author in GetPerson(info.AuthorId) from friends in author.BestFriendIds.SelectFetch(Blog.GetPerson) select ShowList(friends); var result = await fetcher.Fetch(fetch); ; } [TestMethod] public async Task GetNull() { var fetch = from info in FetchPostInfo(3) from author in FetchNullPerson() select author; var result = await fetcher.Fetch(fetch); ; } [TestMethod] public async Task LetNotation_Applicative() { var id = 0; var fetch = from postInfo in FetchPostInfo(id) let id2 = 1 + postInfo.PostId from postInfo2 in FetchPostInfo(id2) select postInfo2.PostTopic + id2; var result = await fetcher.Fetch(fetch); } [TestMethod] public async Task FinalLetNotation_Applicative() { var id = 0; var fetch = from postInfo in FetchPostInfo(id) from postInfo2 in FetchPostInfo(1) let id2 = 1 + postInfo.PostId select postInfo2.PostTopic + id2; var result = await fetcher.Fetch(fetch); Assert.AreEqual("Topic 11", result); } [TestMethod] public async Task FinalLetNotation() { var id = 0; var fetch = from postInfo in FetchPostInfo(id) let x = 3 from postInfo2 in FetchPostInfo(1) let id2 = 1 + postInfo.PostId + 3 select postInfo2.PostTopic + id2 + x; var result = await fetcher.Fetch(fetch); Assert.AreEqual("Topic 143", result); } [TestMethod] public async Task FinalLetNotation_ApplicativeExtended() { var id = 0; var fetch = from postInfo in FetchPostInfo(id) from postInfo2 in FetchPostInfo(1) from postInfo3 in FetchPostInfo(postInfo2.PostId) let id2 = 1 + postInfo.PostId + postInfo3.PostId select postInfo2.PostTopic + id2; var result = await fetcher.Fetch(fetch); } [TestMethod] public async Task MultipleLetNotation_Applicative() { var id = 0; var let = FetchPostInfo(id).Select(info => info.PostId); var fetch = from postInfo in FetchPostInfo(id) let id2 = 1 + postInfo.PostId let id3 = 4 from postInfo2 in FetchPostInfo(id2) select postInfo2.PostTopic + id2 + id3; var result = await fetcher.Fetch(fetch); } [TestMethod] public async Task TwoLatestExample() { var fetch = from latest in FetchTwoLatestPosts() from first in GetPostDetails(latest.Item1) from second in GetPostDetails(latest.Item2) select ShowList(new List { first, second }); var result = await fetcher.Fetch(fetch); } [TestMethod] public async Task DuplicateNestedNames() { var nested = from x in FetchTwoLatestPosts() from z in FetchTwoLatestPosts() from y in GetPostDetails(x.Item1) select $"[Nested: {y.Content}]"; var nested2 = from x in nested from y in nested select $"[Nested2: [ {x}, {y} ]"; var global = new { x = 3 }; var fetch = from x in nested2 from y in nested2 from z in nested let id2 = global.x from n in nested select $"Fetch: [ {x}, {y} ]"; var result = await fetcher.Fetch(fetch); Assert.AreEqual( "Fetch: [ [Nested2: [ [Nested: Post 3], [Nested: Post 3] ], [Nested2: [ [Nested: Post 3], [Nested: Post 3] ] ]", result ); } [TestMethod] public async Task NoNesting() { var fetch = from x in FetchTwoLatestPosts() from y in FetchTwoLatestPosts() from z in FetchTwoLatestPosts() from n in FetchTwoLatestPosts() select $"Fetch: [ {x}, {y} ]"; var result = await fetcher.Fetch(fetch); } [TestMethod] public async Task NestedLet() { var nested = from x in GetPostDetails(3) from y in GetPostDetails(4) select $"[ {x.Content}, {y.Content} ]"; var fetch = from x in nested let id = 3 from y in GetPostDetails(id) select x + y.Content; var result = await fetcher.Fetch(fetch); } [TestMethod] public async Task DuplicateNested() { var nested = from x in GetPostDetails(3) from y in GetPostDetails(4) select $"[ {x.Content}, {y.Content} ]"; var nested2 = from x in nested from z in nested select $"{x}, {z}"; var fetch = from x in nested2 from y in nested select $"{x}, {y}"; var result = await fetcher.Fetch(fetch); } [TestMethod] public async Task TwoLatestExampleAgain() { var fetch = from latest in FetchTwoLatestPosts() from first in GetPostDetails(latest.Item1 + 1) from second in GetPostDetails(latest.Item2 + 2) select new List { first, second }; var result = await fetcher.Fetch(fetch); } [TestMethod] public async Task FetchDetails() { var fetch = GetPostDetails(1); var result = await fetcher.Fetch(fetch); } [TestMethod] public async Task FetchDetails2() { var fetch = from info in FetchPostInfo(2) select new PostDetails(info, "Content"); var result = await fetcher.Fetch(fetch); } [TestMethod] public async Task TransparentAccess() { var fetch = from latest in FetchTwoLatestPosts() from something in FetchPostInfo(1) from somethingElse in FetchPostInfo(2) from three in FetchPostInfo(3) from newPost in FetchPostInfo(latest.Item1) select newPost; var result = await BlogFetch(fetch); } [TestMethod] public async Task Deduplication() { var fetch = from postIds in FetchDuplicatePosts() from details in postIds.SelectFetch(GetPostDetails) select ShowList(details); var result = await BlogFetch(fetch); } private Task BlogFetch(Fetch request) { return fetcher.Fetch(request); } [TestMethod] public async Task WithoutApplicative() { var latest = await BlogFetch(FetchTwoLatestPosts()); var first = await BlogFetch(FetchPostInfo(latest.Item1)); var firstContent = await BlogFetch(FetchPostContent(1)); var second = await BlogFetch(FetchPostInfo(latest.Item2)); var secondContent = await BlogFetch(FetchPostContent(latest.Item2)); } } } ================================================ FILE: HaxlSharp.Test/BindExpressionParseTest.cs ================================================ using System; using Microsoft.VisualStudio.TestTools.UnitTesting; using static HaxlSharp.Internal.Base; namespace HaxlSharp.Test { [TestClass] public class BindExpressionParseTest { [TestMethod] public void ParseBindVar() { var blockNumber = GetBlockNumber("(0) int"); Assert.AreEqual(0, blockNumber); } [TestMethod] public void ParseBigBindVar() { var blockNumber = GetBlockNumber("(7489230) int"); Assert.AreEqual(7489230, blockNumber); } [TestMethod] public void InvalidParse() { try { var blockNumber = GetBlockNumber("((7489230)"); Assert.Fail("No exception thrown."); } catch (ArgumentException) { // pass } } } } ================================================ FILE: HaxlSharp.Test/Blog.cs ================================================ using System; using System.Collections.Generic; using System.Linq; using static HaxlSharp.Internal.Base; using HaxlSharp.Internal; using System.Diagnostics; using System.Threading.Tasks; namespace HaxlSharp.Test { public class Post { public string Title { get; set; } public string Content { get; set; } public int PostId { get; set; } } public class FetchPosts : Returns> { } public class FetchDuplicatePosts : Returns> { } public class GetPerson : Returns { public readonly int PersonId; public GetPerson(int personId) { PersonId = personId; } } public class GetNullPerson : Returns { } public class FetchPostInfo : Returns { public readonly int PostId; public FetchPostInfo(int postId) { PostId = postId; } } public class GetTwoLatestPosts : Returns> { } public class FetchPostContent : Returns { public readonly int PostId; public FetchPostContent(int postId) { PostId = postId; } } public class FetchPostViews : Returns { public readonly int PostId; public FetchPostViews(int postId) { PostId = postId; } } public class Person { public int PersonId; public string Name; public IEnumerable BestFriendIds; public override string ToString() { return $"Person {{ PersonId: {PersonId}, Name: {Name}, BestFriendIds: {BestFriendIds} }}"; } } public static class Blog { public static List Names = new List { "Cherry Greenburg", "Alison Herald", "Michal Zakrzewski", "Chance Kehoe", "Delaine Crago", "Sabina Barrs", "Peg Delosh", "Johnie Wengerd", "Shayne Knauer", "Tyson Dave", "Shandra Hanlin", "Rey Pita", "Jacquelyn Bivona", "Cristal Hornak", "Julieta Kilbane", "Terry Cavin", "Peppa Pig", "Charity Gadsden", "Antione Domingo", "Corazon Benito", "Tianna Bratton", }; public static HaxlFetcher Fetcher() { return FetcherBuilder.New() .FetchRequest>(_ => { return ShowList(Enumerable.Range(0, 10)); }) .FetchRequest>(_ => { return ShowList(Enumerable.Repeat(1, 10)); }) .FetchRequest(req => { var postId = req.PostId; return new PostInfo(postId, DateTime.Today.AddDays(-postId), $"Topic {postId % 3}", (postId * 33) % 20); }) .FetchRequest(req => { return $"Post {req.PostId}"; }) .FetchRequest(req => { return (req.PostId * 33) % 53; }) .FetchRequest(async req => { await Task.Delay(10); return null; }) .FetchRequest(req => { var nameIndex = (req.PersonId * 33) % 20; return new Person { Name = Names.ElementAt(nameIndex), BestFriendIds = ShowList(new List { (nameIndex + 3) % 20, (nameIndex + 5) % 20, (nameIndex + 7) % 20 }), PersonId = req.PersonId }; }) .FetchRequest>(req => { return new Tuple(3, 4); }) .LogWith(log => Debug.WriteLine(log.ToDefaultString())) .Create(); } public static Fetch> FetchTwoLatestPosts() { return new GetTwoLatestPosts().ToFetch(); } public static Fetch> FetchDuplicatePosts() { return new FetchDuplicatePosts().ToFetch(); } public static Fetch> FetchAllPostIds() { return new FetchPosts().ToFetch(); } public static Fetch FetchPostInfo(int postId) { return new FetchPostInfo(postId).ToFetch(); } public static Fetch FetchPostContent(int postId) { return new FetchPostContent(postId).ToFetch(); } public static Fetch GetFirstPostId() { return from posts in GetAllPostInfo() select posts.OrderByDescending(p => p.PostDate).First().PostId; } public static Fetch FetchPostViews(int postId) { return new FetchPostViews(postId).ToFetch(); } public static Fetch> FetchPostAuthorFriends(int postId) { return from info in FetchPostInfo(postId) from author in GetPerson(info.AuthorId) from friends in author.BestFriendIds.SelectFetch(Blog.GetPerson) select ShowList(friends); } public static Fetch FetchNullPerson() { return new GetNullPerson().ToFetch(); } public static Fetch GetPostDetails(int postId) { var x = from info in FetchPostInfo(postId) from content in FetchPostContent(info.PostId) select new PostDetails(info, content); return x; } public static Fetch> GetAllPostInfo() { return from postIds in FetchAllPostIds() from postInfo in postIds.SelectFetch(FetchPostInfo) select postInfo; } public static Fetch GetPerson(int personId) { return new GetPerson(personId).ToFetch(); } public static Fetch> RecentPostContent() { return from posts in GetAllPostInfo() from recentContent in posts.OrderByDescending(p => p.PostDate) .Take(4) .SelectFetch(pi => FetchPostContent(pi.PostId)) select recentContent; } } public class PostDetails { public PostInfo Info; public string Content; public PostDetails(PostInfo info, string content) { Info = info; Content = content; } public override string ToString() { return $"PostDetails {{ Info: {Info}, Content: '{Content}' }}"; } } } ================================================ FILE: HaxlSharp.Test/ExpressionTests.cs ================================================ using Microsoft.VisualStudio.TestTools.UnitTesting; using System; using System.Collections.Generic; using System.Linq; using HaxlSharp.Internal; namespace HaxlSharp.Test { public class Nested { public int x => 99; } [TestClass] public class ExpressionSplitTests { public const int global = 2; public static FetchResult a = new FetchResult(global); public static FetchResult b = new FetchResult(3); public static Func> c = i => new FetchResult(i); public static Func> c2 = (i, i2) => new FetchResult(i); public static Func> d = i => new FetchResult(i); public static Nested nested = new Nested(); public static int CountAt(List split, int i) { return split.ElementAt(i).Expressions.Count(); } public static int SplitCount(List split) { return split.Count(s => s.Expressions.Any(e => e.Match(bind => true, project => false))); } public static int ProjectCount(List split) { return split.Count(s => s.Expressions.Any(e => e.Match(bind => false, project => true))); } public static List Split(Fetch fetch) { var type = fetch.GetType(); Assert.IsTrue(type.GetGenericTypeDefinition() == typeof(Bind<,,>)); return SplitApplicative.SplitBind(fetch.CollectedExpressions, fetch.Initial); } [TestMethod] public void DuplicateVariableNames() { // We've got two variables of different type named 'x'. var nested = from x in new FetchResult("1") // Group 0.1 // split // ========= from za in c(int.Parse(x)) // Group 1.1 from ya in b // Group 1.2 //projection // ========= select ya; // Group 2.1 (Projection) var expression = from x in nested // // split // ========= from z in c(x) // Group 3.1 from y in b // Group 3.2 // split // ========= from w in d(y) // Group 4.1 // projection // ========= select x + y + z + w; // Group 5.1 (Projection) var split = Split(expression); Assert.AreEqual(4, SplitCount(split)); Assert.AreEqual(2, ProjectCount(split)); Assert.AreEqual(1, CountAt(split, 0)); Assert.AreEqual(2, CountAt(split, 1)); Assert.AreEqual(1, CountAt(split, 2)); Assert.AreEqual(2, CountAt(split, 3)); Assert.AreEqual(1, CountAt(split, 4)); } [TestMethod] public void SplitWithApplicativeProject() { var nested = from x in a from y in b select 3; var fetch = from x in nested from y in a from z in b select x + y + z; var split = Split(fetch); Assert.AreEqual(1, SplitCount(split)); } [TestMethod] public void ExpressionTest() { var nested = from xa in new FetchResult(66) // Group 0.1 // split // ========= from za in c(xa) // Group 1.1 from ya in b // Group 1.2 //projection // ========= select xa + ya + za; // Group 2.1 (Projection) var expression = from x in nested // // split // ========= from z in c(x) // Group 3.1 from y in b // Group 3.2 // split // ========= from w in d(y) // Group 4.1 // projection // ========= select x + y + z + w; // Group 5.1 (Projection) var split = Split(expression); Assert.AreEqual(4, SplitCount(split)); Assert.AreEqual(2, ProjectCount(split)); Assert.AreEqual(1, CountAt(split, 0)); Assert.AreEqual(2, CountAt(split, 1)); Assert.AreEqual(1, CountAt(split, 2)); Assert.AreEqual(2, CountAt(split, 3)); Assert.AreEqual(1, CountAt(split, 4)); } [TestMethod] public void ExpressionTest2() { var expression = from x in a from y in new FetchResult(nested.x) // split from z in c2(x, y) select x + y + z; var split = Split(expression); Assert.AreEqual(2, SplitCount(split)); Assert.AreEqual(2, CountAt(split, 0)); Assert.AreEqual(1, CountAt(split, 1)); } [TestMethod] public void ExpressionTest3() { var expression = from x in a // split from y in c(x) from z in c(x) select x + y + z; var split = Split(expression); Assert.AreEqual(2, SplitCount(split)); Assert.AreEqual(1, CountAt(split, 0)); Assert.AreEqual(2, CountAt(split, 1)); } [TestMethod] public void ExpressionTest4() { var expression = from f in a from x in a // split from y in c(x) from z in c(nested.x) select x + y + z; var split = Split(expression); Assert.AreEqual(2, SplitCount(split)); Assert.AreEqual(2, CountAt(split, 0)); Assert.AreEqual(2, CountAt(split, 1)); } [TestMethod] public void ExpressionTest5() { var expression = from x in a from z in c(nested.x) // split from y in c(x + 3) select x + y + z; var split = Split(expression); Assert.AreEqual(2, SplitCount(split)); Assert.AreEqual(2, CountAt(split, 0)); Assert.AreEqual(1, CountAt(split, 1)); } [TestMethod] public void ExpressionTest6() { var expression = from z in c(nested.x) from x in a // split from y in c(x + 3) select x + y + z; var split = Split(expression); Assert.AreEqual(2, SplitCount(split)); Assert.AreEqual(2, CountAt(split, 0)); Assert.AreEqual(1, CountAt(split, 1)); } [TestMethod] public void LetTest() { var expression = from x in a let q = x + 3 from z in c(q) from y in c(q + 3) select x + y + z; var split = Split(expression); } [TestMethod] public void NestedQuery() { var expression = from x in (from x in a select x + 1) from y in b //split from z in c(x) from w in d(y) select x + y + z + w; var split = Split(expression); } [TestMethod] public void SequenceRewrite() { var list = Enumerable.Range(0, 10); Func> mult10 = x => new FetchResult(x * 10); var expression = from x in new FetchResult>(list) from multiplied in x.SelectFetch(mult10) from added in x.SelectFetch(num => new FetchResult(num + 1)) select added.Concat(multiplied); var split = Split(expression); Assert.AreEqual(2, SplitCount(split)); Assert.AreEqual(1, CountAt(split, 0)); Assert.AreEqual(2, CountAt(split, 1)); } [TestMethod] public void SequenceRewriteConcurrent() { var list = Enumerable.Range(0, 1000); Func> mult10 = x => new FetchResult(x * 10); var expression = from x in new FetchResult>(list) from multiplied in x.SelectFetch(mult10) from added in x.SelectFetch(num => new FetchResult(num)) select added.Concat(multiplied); var split = Split(expression); Assert.AreEqual(2, SplitCount(split)); Assert.AreEqual(1, CountAt(split, 0)); Assert.AreEqual(2, CountAt(split, 1)); } [TestMethod] public void OneLiner() { var oneLine = from x in new FetchResult(3) select x + 1; } [TestMethod] public void SelectTest() { var number = new FetchResult(3); var plusOne = number.Select(num => num + 1); var plusTwo = plusOne.Select(num => num + 1); } } } ================================================ FILE: HaxlSharp.Test/HaxlSharp.Test.csproj ================================================  Debug AnyCPU {B62BE199-4FAB-41BF-BB52-A194E7E8106D} Library Properties HaxlSharp.Test HaxlSharp.Test v4.5.2 512 {3AC096D0-A1C2-E12C-1390-A8335801FDAB};{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC} 10.0 $(MSBuildExtensionsPath32)\Microsoft\VisualStudio\v$(VisualStudioVersion) $(ProgramFiles)\Common Files\microsoft shared\VSTT\$(VisualStudioVersion)\UITestExtensionPackages False UnitTest true full false bin\Debug\ DEBUG;TRACE prompt 4 pdbonly true bin\Release\ TRACE prompt 4 False {56487eb5-c699-4eaa-b384-c6f9e64635c5} HaxlSharp.Core {4d87351f-eee6-4361-b889-d8217eb314aa} HaxlSharp.Fetcher False False False False ================================================ FILE: HaxlSharp.Test/MockData.cs ================================================ using System; using System.Collections.Generic; using System.Linq; using System.Text; using System.Threading.Tasks; namespace HaxlSharp.Test { public class PostInfo { public readonly int PostId; public readonly DateTime PostDate; public readonly string PostTopic; public readonly int AuthorId; public PostInfo(int postId, DateTime postDate, string postTopic, int authorId) { PostId = postId; PostDate = postDate; PostTopic = postTopic; AuthorId = authorId; } public override string ToString() { return $"PostInfo {{ Id: {PostId}, Date: {PostDate.ToShortDateString()}, Topic: '{PostTopic}'}}"; } } } ================================================ FILE: HaxlSharp.Test/Properties/AssemblyInfo.cs ================================================ using System.Reflection; using System.Runtime.CompilerServices; using System.Runtime.InteropServices; // General Information about an assembly is controlled through the following // set of attributes. Change these attribute values to modify the information // associated with an assembly. [assembly: AssemblyTitle("HaxlSharp.Test")] [assembly: AssemblyDescription("")] [assembly: AssemblyConfiguration("")] [assembly: AssemblyCompany("")] [assembly: AssemblyProduct("HaxlSharp.Test")] [assembly: AssemblyCopyright("Copyright © 2016")] [assembly: AssemblyTrademark("")] [assembly: AssemblyCulture("")] // Setting ComVisible to false makes the types in this assembly not visible // to COM components. If you need to access a type in this assembly from // COM, set the ComVisible attribute to true on that type. [assembly: ComVisible(false)] // The following GUID is for the ID of the typelib if this project is exposed to COM [assembly: Guid("b62be199-4fab-41bf-bb52-a194e7e8106d")] // Version information for an assembly consists of the following four values: // // Major Version // Minor Version // Build Number // Revision // // You can specify all the values or you can default the Build and Revision Numbers // by using the '*' as shown below: // [assembly: AssemblyVersion("1.0.*")] [assembly: AssemblyVersion("0.1.*")] ================================================ FILE: HaxlSharp.sln ================================================  Microsoft Visual Studio Solution File, Format Version 12.00 # Visual Studio 2012 VisualStudioVersion = 14.0.24720.0 MinimumVisualStudioVersion = 10.0.40219.1 Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "HaxlSharp.Test", "HaxlSharp.Test\HaxlSharp.Test.csproj", "{B62BE199-4FAB-41BF-BB52-A194E7E8106D}" EndProject Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "HaxlSharp.Core", "HaxlSharp.Core\HaxlSharp.Core.csproj", "{56487EB5-C699-4EAA-B384-C6F9E64635C5}" EndProject Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "HaxlSharp.Fetcher", "HaxlSharp.Fetcher\HaxlSharp.Fetcher.csproj", "{4D87351F-EEE6-4361-B889-D8217EB314AA}" EndProject Global GlobalSection(SolutionConfigurationPlatforms) = preSolution Debug|Any CPU = Debug|Any CPU Release|Any CPU = Release|Any CPU EndGlobalSection GlobalSection(ProjectConfigurationPlatforms) = postSolution {4D87351F-EEE6-4361-B889-D8217EB314AA}.Debug|Any CPU.ActiveCfg = Debug|Any CPU {4D87351F-EEE6-4361-B889-D8217EB314AA}.Debug|Any CPU.Build.0 = Debug|Any CPU {4D87351F-EEE6-4361-B889-D8217EB314AA}.Release|Any CPU.ActiveCfg = Release|Any CPU {4D87351F-EEE6-4361-B889-D8217EB314AA}.Release|Any CPU.Build.0 = Release|Any CPU {56487EB5-C699-4EAA-B384-C6F9E64635C5}.Debug|Any CPU.ActiveCfg = Debug|Any CPU {56487EB5-C699-4EAA-B384-C6F9E64635C5}.Debug|Any CPU.Build.0 = Debug|Any CPU {56487EB5-C699-4EAA-B384-C6F9E64635C5}.Release|Any CPU.ActiveCfg = Release|Any CPU {56487EB5-C699-4EAA-B384-C6F9E64635C5}.Release|Any CPU.Build.0 = Release|Any CPU {B62BE199-4FAB-41BF-BB52-A194E7E8106D}.Debug|Any CPU.ActiveCfg = Debug|Any CPU {B62BE199-4FAB-41BF-BB52-A194E7E8106D}.Debug|Any CPU.Build.0 = Debug|Any CPU {B62BE199-4FAB-41BF-BB52-A194E7E8106D}.Release|Any CPU.ActiveCfg = Release|Any CPU {B62BE199-4FAB-41BF-BB52-A194E7E8106D}.Release|Any CPU.Build.0 = Release|Any CPU EndGlobalSection GlobalSection(SolutionProperties) = preSolution HideSolutionNode = FALSE EndGlobalSection EndGlobal ================================================ FILE: LICENCE ================================================ MIT License Copyright (c) 2016 Joash Chong Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. ================================================ FILE: README.md ================================================ # HaxlSharp A C# implementation of [Haxl](https://github.com/facebook/Haxl) for composable data fetching with automatic concurrency and request deduplication. Not affiliated with Facebook in any way! Table of Contents ================= * [Quick start](#quick-start) * [What's wrong with async/ await?](#whats-wrong-with-async-await) * [Composing async methods](#composing-async-methods) * [What's wrong with Task.WhenAll/ Promise.all?](#whats-wrong-with-taskwhenall-promiseall) * [Haxl: reclaiming the sequential abstraction](#haxl-reclaiming-the-sequential-abstraction) * [Composing requests](#composing-requests) * [Request deduplication](#request-deduplication) * [Implementation details](#implementation-details) * [Integration](#integration) * [Defining requests](#defining-requests) * [Using the requests](#using-the-requests) * [Handling requests](#handling-requests) * [Implementing your own fetcher](#implementing-your-own-fetcher) * [Why would you want to implement your own fetcher/ caching strategy?](#why-would-you-want-to-implement-your-own-fetcher-caching-strategy) * [Fetcher](#fetcher) * [Caching](#caching) * [Limitations](#limitations) * [Speed](#speed) * [Anonymous types](#anonymous-types) * [Applicative Do](#applicative-do) * [It's a giant hack](#its-a-giant-hack) ## Quick start Install from nuget: [https://www.nuget.org/packages/HaxlSharp](https://www.nuget.org/packages/HaxlSharp) Before you can use the library, you'll need to write a thin layer to get your existing data sources integrated with HaxlSharp- see the [Integration](#integration) section, or you can check out an example application using HaxlFetch [here](https://github.com/joashc/HaxlSharpDemo). Once that's done, you can write your data fetches in a sequential way, and the framework will automatically perform requests as concurrently as possible, and do request deduplication. ## What's wrong with async/ await? Async/ await is great for writing sequential-looking code when you're only waiting for a single asynchronous request at a time, allowing us to write code without worrying about asynchronicity. But we often want to combine information from multiple data sources, like different calls on the same API, or multiple remote APIs. The async/ await abstraction breaks down in these situations (and Javascript's async/await is no different). To illustrate, let's say we have a blogging site, and a post's metadata and content are retrieved using separate API calls. We could use async/ await to fetch both these pieces of information: ```cs public async Task GetPostDetails(int postId) { var postInfo = await FetchPostInfo(postId); var postContent = await FetchPostContent(postId); return new PostDetails(postInfo, postContent); } ``` Here, we're making two successive `await` calls, which means the execution will be suspended at the first request- `FetchPostInfo`- and only begin executing the second request- `FetchPostContent`- once the first request has completed. But fetching `FetchPostContent` doesn't require the result of `FetchPostInfo`, which means we could have started both these requests concurrently! The "correct" way to write it is: ```cs var postInfoTask = FetchPostInfo(postId); var postContentTask = FetchPostContent(postId); return new PostDetails(await postInfo, await postContent); ``` But now we are dealing with tasks instead of their values; it's up to the programmer to ensure the task is `await`ed as late as possible. Async/ await is a good abstraction for *asynchronous* code, but writing *concurrent* code requires us to mix code that describes *what* we want to fetch with *how* we want to fetch it. ### Composing async methods To make matters worse, we can easily call our inefficient `GetPostDetails` method in a way that compounds the oversequentialization: ```cs public async Task LatestPostContent() { var latest = await GetTwoLatestPostIds(); var first = await GetPostDetails(latest.Item1); var second = await GetPostDetails(latest.Item2); return new List{first, second}; } ``` This code will sequentially execute four calls that could have been executed concurrently! We should actually write our code like this: ```cs var latest = await GetTwoLatestPostIds(); var first = GetPostDetails(latest.Item1); var second = GetPostDetails(latest.Item2); return new List { await first, await second }; ``` ### What's wrong with `Task.WhenAll`/ `Promise.all`? We can manually add concurrency by giving up sequential-looking code that doesn't make a distinction between async values and "normal" values. In practice, this means dealing with both tasks and their `await`ed values, and sprinkling our code with `Task.WhenAll`. But hang on, async/await was designed to solve these problems: - Writing asynchronous code is error-prone - Asynchronous code obscures the meaning of what we're trying to achieve - Programmers are bad at reasoning about asynchronous code Giving up our sequential abstraction means these exact problems have reemerged in the context of concurrency! - Writing **concurrent** code is error-prone - **Concurrent** code obscures the meaning of what we're trying to achieve - Programmers are bad at reasoning about **concurrent** code ## Haxl: reclaiming the sequential abstraction Haxl allows us to write code that *looks* like it operates sequentially on "normal values", but is capable of being analyzed to determine the requests we can fetch concurrently, and then automatically batch these requests into a list. This has a number of advantages over async/await and `Task.WhenAll`: - We can write code that uses the results of asynchronous requests, without the risk of losing concurrency. - Multiple requests to a single endpoint can be batched and handled more efficiently- for example, multiple concurrent requests to an SQL database could be rewritten into a single `SELECT` statement. - We only fetch duplicate requests once, even if the duplicate requests are started concurrently- something we can't achieve with `async` or `Task.WhenAll`. - Only fetching data once ensures data remains consistent within a request. Taken together, these advantages leave programmers free to compose complex data fetches, without worrying about concurrency or duplication. It also lessens the need to traverse large parts of the stack to write specialized data fetching methods. Only a small number of "primitive requests" need to be written across the stack; the rest can be composed from these primitives as necessary. Let's rewrite `GetPostDetails` using HaxlSharp: ```cs Fetch GetPostDetails(int postId) => from info in FetchPostInfo(postId) from content in FetchPostContent(postId) select new PostDetails(info, content); ``` The framework can automatically work out that these calls can be parallelized. Here's the debug log from when we fetch `GetPostDetails(1)`: ``` ==== Batch ==== Fetched 'info': PostInfo { Id: 1, Date: 10/06/2016, Topic: 'Topic 1'} Fetched 'content': Post 1 ==== Result ==== PostDetails { Info: PostInfo { Id: 1, Date: 10/06/2016, Topic: 'Topic 1'}, Content: 'Post 1' } ``` Both requests were automatically placed in a single batch and fetched concurrently! ### Composing requests Let's compose our new `GetPostDetails` function: ```cs Fetch> GetLatestPostDetails() => from latest in FetchTwoLatestPostIds() // We must wait here from first in GetPostDetails(latest.Item1) from second in GetPostDetails(latest.Item2) select new List { first, second }; ``` If we fetch this, we get: ``` ==== Batch ==== Fetched 'latest': (0, 1) ==== Batch ==== Fetched 'content': Post 1 Fetched 'info': PostInfo { Id: 1, Date: 10/06/2016, Topic: 'Topic 1'} Fetched 'content': Post 0 Fetched 'info': PostInfo { Id: 0, Date: 11/06/2016, Topic: 'Topic 0'} ==== Result ==== [ PostDetails { Info: PostInfo { Id: 0, Date: 11/06/2016, Topic: 'Topic 0'}, Content: 'Post 0' }, PostDetails { Info: PostInfo { Id: 1, Date: 10/06/2016, Topic: 'Topic 1'}, Content: 'Post 1' } ] ``` The framework has worked out that we have to wait for the first call's result before continuing, because we rely on this result to execute our subsequent calls. But the subsequent two calls only depend on `latest`, so once `latest` is fetched, they can both be fetched concurrently! Note that we made two parallelizable calls to `GetPostDetails`, which is itself made up of two parallelizable requests. These requests were "pulled out" and placed into a single batch of four concurrent requests. Let's see what happens if we rewrite `GetPostDetails` so that it must make two sequential requests: ```cs Fetch GetPostDetails(int postId) => from info in FetchPostInfo(postId) // We need to wait for the result of info before we can get this id! from content in FetchPostContent(info.Id) select new PostDetails(info, content); ``` now when we fetch `GetLatestPostDetails`, we get: ``` ==== Batch ==== Fetched 'latest': (0, 1) ==== Batch ==== Fetched 'info': PostInfo { Id: 1, Date: 10/06/2016, Topic: 'Topic 1'} Fetched 'info': PostInfo { Id: 0, Date: 11/06/2016, Topic: 'Topic 0'} ==== Batch ==== Fetched 'content': Post 1 Fetched 'content': Post 0 ==== Result ==== [ PostDetails { Info: PostInfo { Id: 0, Date: 11/06/2016, Topic: 'Topic 0'}, Content: 'Post 0' }, PostDetails { Info: PostInfo { Id: 1, Date: 10/06/2016, Topic: 'Topic 1'}, Content: 'Post 1' } ] ``` The `info` requests within `GetPostDetails` can be fetched with just the result of `latest`, so they were batched together. The remaining `content` batch can resume once the `info` batch completes. ### Request deduplication Because we lazily compose our requests, we can keep track of every subrequest within a particular request, and only fetch a particular subrequest once, even if they're part of the same batch. Let's say that each post has an author, and each author has three best friends. We could fetch the friends of the author of a given post like this: ```cs Fetch> PostAuthorFriends(int postId) => from info in FetchPostInfo(postId) from author in GetPerson(info.AuthorId) from friends in author.BestFriendIds.SelectFetch(GetPerson) select friends; ``` Here, we're using `SelectFetch`, which lets us run a request for every item in a list, and get back the list of results. (`SelectFetch` has the signature `[a] -> (a -> Fetch a) -> Fetch [a]`- basically a monomorphic `sequenceA` to Haskellers). Let's fetch `PostAuthorFriends(3)`: ``` ==== Batch ==== Fetched 'info': PostInfo { Id: 3, Date: 8/06/2016, Topic: 'Topic 0'} ==== Batch ==== Fetched 'author': Person { PersonId: 19, Name: Johnie Wengerd, BestFriendIds: [ 10, 12, 14 ] } ==== Batch ==== Fetched 'friends[2]': Person { PersonId: 14, Name: Michal Zakrzewski, BestFriendIds: [ 5, 7, 9 ] } Fetched 'friends[0]': Person { PersonId: 10, Name: Shandra Hanlin, BestFriendIds: [ 13, 15, 17 ] } Fetched 'friends[1]': Person { PersonId: 12, Name: Peppa Pig, BestFriendIds: [ 19, 1, 3 ] } ==== Result ==== [ Person { PersonId: 10, Name: Shandra Hanlin, BestFriendIds: [ 13, 15, 17 ] }, Person { PersonId: 12, Name: Peppa Pig, BestFriendIds: [ 19, 1, 3 ] }, Person { PersonId: 14, Name: Michal Zakrzewski, BestFriendIds: [ 5, 7, 9 ] } ] ``` Calling `.SelectFetch(GetPerson)` on a list of three `PersonId`s gets us a list of three `Person` objects. Each item in the list was fetched in a single concurrent batch. Now let's see how we handle duplicate requests: ```cs from ids in FetchTwoLatestPosts() from friends1 in PostAuthorFriends(ids.Item1) from friends2 in PostAuthorFriends(ids.Item2) select friends1.Concat(friends2); ``` Fetching this gives us: ``` ==== Batch ==== Fetched 'ids': (3, 4) ==== Batch ==== Fetched 'info': PostInfo { Id: 3, Date: 8/06/2016, Topic: 'Topic 0'} Fetched 'info': PostInfo { Id: 4, Date: 7/06/2016, Topic: 'Topic 1'} ==== Batch ==== Fetched 'author': Person { PersonId: 12, Name: Peppa Pig, BestFriendIds: [ 19, 1, 3 ] } Fetched 'author': Person { PersonId: 19, Name: Johnie Wengerd, BestFriendIds: [ 10, 12, 14 ] } ==== Batch ==== Fetched 'friends[0]': Person { PersonId: 10, Name: Shandra Hanlin, BestFriendIds: [ 13, 15, 17 ] } Fetched 'friends[2]': Person { PersonId: 3, Name: Corazon Benito, BestFriendIds: [ 2, 4, 6 ] } Fetched 'friends[1]': Person { PersonId: 1, Name: Cristal Hornak, BestFriendIds: [ 16, 18, 0 ] } Fetched 'friends[2]': Person { PersonId: 14, Name: Michal Zakrzewski, BestFriendIds: [ 5, 7, 9 ] } ==== Result ==== [ Person { PersonId: 10, Name: Shandra Hanlin, BestFriendIds: [ 13, 15, 17 ] }, Person { PersonId: 12, Name: Peppa Pig, BestFriendIds: [ 19, 1, 3 ] }, Person { PersonId: 14, Name: Michal Zakrzewski, BestFriendIds: [ 5, 7, 9 ] }, Person { PersonId: 1, Name: Cristal Hornak, BestFriendIds: [ 16, 18, 0 ] }, Person { PersonId: 19, Name: Johnie Wengerd, BestFriendIds: [ 10, 12, 14 ] }, Person { PersonId: 3, Name: Corazon Benito, BestFriendIds: [ 2, 4, 6 ] } ] ``` Because Peppa Pig and Johnie Wengerd are each other's best friends, we don't need to fetch them again when we're fetching their best friends. The fourth batch, where the best friends of Peppa and Johnie are fetched, only contains four requests, but the results are still correctly compiled into a list of six best friends. This is also helpful for consistency; even though data can change during a fetch, we can still ensure that we don't get multiple versions of the same data within a single fetch. ## Implementation details The [original paper](http://community.haskell.org/~simonmar/papers/haxl-icfp14.pdf) gives a good overview of Haxl. Some differences between the C# and Haskell version are documented [here](http://joashc.github.io/posts/2016-06-11-haxlsharp.html). ## Integration The default API is similar to ServiceStack's, but it's straightforward to implement your own API if this one is not to your taste. You can implement your own API to HaxlSharp by just installing the `HaxlSharp.Core` package, instead of the `HaxlSharp` package, and implementing your own fetcher/ caching strategy. See [Implementing your own fetcher](#implementing-your-own-fetcher) for more details. ### Defining requests You can define requests with POCOs; just annotate their return type like so: ```cs public class FetchPostInfo : Returns { public readonly int PostId; public FetchPostInfo(int postId) { PostId = postId; } } ``` ### Using the requests This library operates on `Fetch<>` objects, so we write functions that create `Returns<>` objects and then call `ToFetch` on them: ```cs Fetch GetPostInfo(int postId) => new GetPostInfo(postId).ToFetch(); Fetch GetPostContent(int postId) => new GetPostContent(postId).ToFetch(); ``` Now we can compose any function that returns a `Fetch<>`, and they'll be automatically batched as much as possible: ```cs Fetch GetPostDetails(int postId) => from info in GetPostInfo(postId) from content in GetPostContent(postId) select new PostDetails(info, content); Fetch> RecentPostDetails() => from postIds in GetAllPostIds() from postDetails in postIds.Take(10).SelectFetch(GetPostDetails) select postDetails; ``` ### Handling requests Of course, the data must come from somewhere, so we must create handlers for every request type. Handlers are just functions from the request type to the response type. Register these functions to create a `Fetcher` object: ```cs var fetcher = FetcherBuilder.New() .FetchRequest>(_ => _postApi.GetAllPostIds()) .FetchRequest(req => _postApi.GetPostInfo(req.PostId)) .FetchRequest(req => _userApi.GetUser(req.UserId)) .Create(); ``` This object can be injected wherever you want to resolve a `Fetch` into an `A`: ```cs Fetch> getPopularContent = from postIds in GetAllPostIds() from views in postIds.SelectFetch(GetPostViews) from popularPosts in views.OrderByDescending(v => v.Views) .Take(5) .SelectFetch(v => GetPostContent(v.Id)) select popularPosts; IEnumerable popularContent = await fetcher.Fetch(getPopularContent); ``` We should work within the `Fetch<>` monad as much as possible, and only resolve the final `Fetch<>` when absolutely necessary. This ensures the framework performs the fetches in the most efficient way. ### Implementing your own fetcher This library comes with a default fetching and caching strategy, but it's possible to implement your own. #### Why would you want to implement your own fetcher/ caching strategy? - You think there's too much boilerplate with the default fetcher. The current implementation is optimized to make it easy to integrate with existing request objects, at the cost of slightly more boilerplate. - You don't want the overhead of serializing every request object to get a cache key, and/ or already have a way to create cache keys from your request objects. - You can do something clever with a batch of requests- you might bundle them up and send them to a particular server, for example. - You want to inject default values of failed requests #### Fetcher The fetcher interface mainly requires you to implement: ```cs Task FetchBatch(IEnumerable requests); ``` A blocked request contains a request object and a `TaskCompletionSource`, which allows us to manually create and resolve `Task` objects- think Javascript promises. You'll want to map the list of blocked requests to a list of `Task`s, and manually resolve each one with the result of its respective request. Then you just return `Task.WhenAll` on the list of tasks! #### Caching To customize the caching behaviour, you need to implement a function that returns a cache key for a request: ```cs string ForRequest(Returns request); ``` Note that this is not traditional caching- it's intrarequest caching used for request deduplication and consistency. You'll still want to have a traditional caching layer. ## Limitations This library is still in its very early stages! Here is a very incomplete list of its limitations: ### Speed This is the most important one: it's still very unoptimized, so it adds an overhead that might not pay for itself! Queries written in the `Fetch` monad are actually treated as expression trees so they can be analysed to determine their dependencies. The expression trees are rewritten to maximise concurrency, and then compiled. Unfortunately, expression tree compilation is *slow* in C\#! This makes `SelectFetch` very inefficient on larger lists, because it compiles multiple expression trees for each item in the list. The Haskell version seems to have a similar asymptotic complexity, but with a *much* smaller constant. (Current plan for optimizing: instead of compiling an expression tree for each item in the list `[a]`, I could compile the expression `a -> Expression` once, and then plug `a` into this compiled expression. We'll see how this works out.) ### Anonymous types It's not recommended to return anonymous types from your `Fetch` functions, unless you want your functions to fail unpredictably. C\# uses anonymous types internally for query expressions, which is alright in the case of [transparent identifiers](http://joashc.github.io/posts/2016-03-17-select-many-weird.html) because they're tagged with a special name only available to the compiler, but `let` statements are translated into plain old anonymous types that are indistinguishable from the ones you could type in manually: ```cs from a in x from b in y select new {a, b}; ``` There a few checks in place so anonymous types won't fail in all circumstances, but unless you want to memorize this list and ensure your expression doesn't meet all of these criteria: - Expression body is `ExpressionType.New` - Creates an anonymous type - Anonymous type has exactly two properties - First property of anonymous type is the same as the first parameter of the expression ...you're better off just not using anonymous types. ### Applicative Do We're currently using a simplified version of the `ApplicativeDo` algorithm in GHC, so a query expression like: ```cs from x in a from y in b(x) from z in c ``` is executed in three batches, even though `a` and `c` could be started concurrently. ### It's a giant hack The C# language spec goes to the effort of saying that the implementation details of query expression rewriting and scoping are *not* part of the specification, and different implementations of C# can do query rewriting/scoping differently. Of course, this library builds heavily on the internal, unspecified implementation details of query rewriting and scoping, so it's possible that the C\# team could reimplement it and break the library. I think the C# team kept transparent identifiers, etc. out of the spec because they knew they were a bit of a hack to get the desired transitive scoping behaviour, which actually *is* part of the spec. So this library is a hack raised upon a hack... but it's called HaxlSharp, so at least the name is apt. ================================================ FILE: buildNuget.bat ================================================ move /Y nuget\*.nupkg nuget\previous nuget pack HaxlSharp.Core\HaxlSharp.Core.csproj -Prop Configuration=Release -OutputDirectory nuget\ nuget pack HaxlSharp.Fetcher\HaxlSharp.Fetcher.csproj -Prop Configuration=Release -IncludeReferencedProjects -OutputDirectory nuget\ pause