\n \n\tExtended Memory Semantics (EMS) complements serial programming\n\tmodels with \n\ttransactional and other fine-grained synchronization capabilities\n\tto support parallel programming.\n \n
\n Much of the challenge in implementing distributed and parallel\n programs derives from finding, marshaling, and synchronizing\n data. Extended Memory Semantics (EMS) unifies these tasks into a\n single programming and execution model. EMS implements a shared\n address space with a rich set of primitives for parallel access of data\n structures. It is not a source of parallelism itself, instead it\n complements other parallel programming models and integrates shared\n memory data access and synchronization.\n
\n EMS leverages existing tool chains instead of replacing them and is \n compatible with legacy applications, libraries, frameworks,\n operating systems, and hardware. \n Because EMS represents \n memory and not processes, it may persist independently of any\n application, and it's state may be replicated, archived, or forked.\n Applications may attach and detach from the memory in much the\n same way applications use a shared database or filesystem.\n
\n\n
\n
Synchronization As a Property of the Data, Not a Duty for Tasks\n
\n\n
\n EMS internally stores tags that are used for synchronization of\n user data, allowing synchronization to happen independently of\n the number or kind of processes accessing the data. The tags\n can be thought of as being in one of three states, Empty,\n Full, or Read-Only, and the EMS primitives enforce\n atomic access through automatic state transitions.\n \n \n \n\t
EMS Data Tag Transitions & Atomic operations:\n\tF=Full, E=Empty, X=Don't Care, RW=Readers-Writer lock (# of current readers)\n\tCAS=Compare-and-Swap, FAA=Fetch-and-Add\n \n
\n The function name readFE means \"Read when full and mark empty\",\n writeEF means \"Write when empty and mark full\",\n writeXF means \"Write unconditionally and mark\n full\", etc. In the most simple case, full-empty tags are used\n to block readers until memory is marked full by a writer thread\n that itself blocks until the memory is marked empty. This is\n effectively a dataflow or producer-consumer execution model that\n may involve any number of producers and consumers.\n
\n \n \n\t
EMS memory is an array of JSON primitive values\n\t(Number, Boolean, String, or Undefined) accessed using atomic\n\toperators and/or transactional memory. Safe parallel access\n\tis managed by passing through multiple gates: First mapping a\n\tkey to an index, then accessing user data protected by EMS\n\ttags, and completing the whole operation atomically.\n \n
\n The EMS array may be indexed directly using an integer, or using a key\n mapping of any primitive type. When a map is used, the key and data\n itself are updated atomically.\n
\n The full-empty primitives are\n used construct other thread-safe data types such as \n atomic read-modify-write operations and Transactional Memory (TM).\n
\n\n\n
\n Principles of Operation\n
\n\n
\n When the require('ems')(...) statement is executed by a program,\n EMS first creates a shared memory region to rendezvous and\n communicate with other EMS threads, then,\n using the built-in\n \n fork primitives, \n creates the additional threads executing \n using one of two execution models: fork-join or Bulk Synchronous Parallel (BSP).\n BSP invokes the same script as the master thread (found in process.argv[2]),\n whereas fork-join execution invokes parallel region around a function.\n
\n Under BSP, all threads execute the entire program unless statements are explicitly skipped.\n Fork-join parallelism has a single entry point and executes sequentially\n until a parallel region is started with ems.parallel( func ).\n \n \n \n\tFork-Join parallelism follows the traditional single-threaded execution\n\tmodel until a parallel region where threads are dynamically added to\n\tperform iterations of a loop. Under BSP parallelism every thread\n\tenters the program at the main entry point. \n \n
\n Fork-join creates parallel regions\n much like OpenMP's #pragma omp parallel directive.\n Under BSP, all threads enter the main program and execute all\n statements, synchronizing at barriers.\n\n
\n In addition to ordinary sequential loops,\n within a parallel region ems.parForEach( func )\n loops distribute iterations among the threads\n using several load balancing scheduling options.\n
\n The master thread preserves all the characteristics and capabilities of\n an ordinary \n \n job, and all legacy applications, modules, packages, frameworks, and test apparatus\n will work normally. Software that does not use EMS is not affected by it's presence.\n
\n Atomic operations like compare-and-swap (CAS) and fetch-and-add (FAA)\n that are typically non-blocking will block if the full/empty\n tag is set to empty.\n Stack/queue operators are deadlock free,\n blocking operations and should be thought of as thread-safe but not concurrent.\n EMS transactions are also deadlock free and support element-level locking\n for the highest possible currency.\n
\n Dataflow programs directly manipulating the full/empty tags\n may deadlock if a program attempts to re-acquire a lock\n already held, or acquire locks in a different order than other threads.\n
\n EMS programs may be run with any number of threads, including\n single threaded and over-subscribed. \n
\n\n \n
\n \n \n\t
\n\tA logical overview of what program statements cause threads to be created\n\tand how shared data is referenced.\n \n
\n\n\n
Performance
\n
\n These experiments were run on an Amazon EC2 instance: \n cr1.8xlarge: 244 GiB memory, 88 EC2 Compute Units, 240 GB of local instance storage, 64-bit platform, 10 Gigabit Ethernet\n
\n
\n \n \n\tUsing the general transactional memory capabilities to process\n\trandomly generated operations stored in a shared work queue.\n \n
\n\t\n \n\tTransaction processing but generating the work from a loop instead of reading it from a shared queue.\n \n
\n \n \n\tWord Count of documents from Project Gutenberg in a variety of languages. Average document was\n\tabout 250kb in length.\n \n
\n\n\n
Built-In Composed Operations and Parallel Data Structures
\n
\n High-level data abstractions can be constructed from the EMS primitives,\n and EMS itself \n composes the primitives to implement transactional memory (TM), stacks, and queues.\n User defined composed operations can be added to EMS classes just as new methods\n are added to other JavaScript objects.\n
\n\n\n\n
Transactional Memory
\n
\n Transactional Memory (TM) provides atomic access to multiple shared objects\n in a manner similar to transactional databases. EMS implements mutual exclusion\n on specific data elements using the Full/Empty tags,\n and shared read-only access with a multiple readers-single writer tag.\n\n
\n\n\n\n\n
Stacks and Queues
\n
\n Parallel-safe stacks and queues are built-in intrinsics based on Full/Empty tags.\n Stacks and queues are by definition serial data structures and do\n not support any concurrency. Although highly efficient, a shared\n resource like these can become a hot-spot when dozens of threads compete\n for access.\n
\n\n\n\n\n\n
\n Types of Parallelism\n
\n\n
\n
\n\t\n\t \n\t \n\t EMS Data Tag Transitions - The four data element states \n\t and the intrinsic EMS atomic operations to transition\n\t between them.\n\t \n\t\n
\n \n\n\n
\n Why Shared Memory Parallelism?\n
\n\n
\n
\n\t\n\t \n\t \n\t Multithreading complements other forms of parallelism and can be\n\t combined with other forms of concurrency for multiplicative benefits.\n\n\t \n\t\n
\n \n\n\n
\n Contrary Notions of Strong & Weak Scaling\n
\n\n
\n
\n\t
\n\t
\n
\n \n
\n
\n
\n\t\tStrong Scaling \n\t\t\n
\n
\n
\n
\n\t\tWeak Scaling \n\t\t\n
\n
\n\t
\n\n\t
\n
\n Scaling Goal\n
\n
\n Solve the same problem, only faster\n
\n
\n Solve a bigger problem in the same amount of time\n
\n\t
\n\n\t
\n
\n Problem Size\n
\n
\n Stays constant while number of processors increases\n
\n
\n Grows with the number of processors\n
\n\t
\n\n\t
\n
\n Scaling is limited by\n
\n
\n Inter-process communication\n
\n
\n Problem size\n
\n\t
\n\n\t
\n
\n Resiliency\n
\n
\n Single failure causes entire job to fail,\n SLAs achieved through efficient checkpoint-restart.\n
\n
\n Failed sub-tasks are detected and retried. SLAs achieved through fault resiliency.\n
\n Historical Precedents for Data-Centric Multithreading\n
\n
\n\n EMS builds on three experimental computer architectures from the\n 1980's: the NYU Ultracomputer, the MIT J-Machine, and the Tera Multi-Threaded Architecture (MTA).\n\n Specifically, the Ultra introduced combining networks as a basic\n architectural feature,\n the J-Machine made moving a task to data as easy as moving data to the processor,\n and the MTA used massive multithreading to mask latency and\n had fine-grained synchronization associated with the data, not tasks.\n
\n\n\n\n\n
\n Bugs\n
\n
\n The old data argument for CAS requires a deep compare -- this is possible but\n entirely compatible with JS semantics which compares type, not contents.\n
\n Similarly, FAA does not implement add on object type data, but extending the\n semantics of JS would make this sensible\n
\n\n\n\n\n
\n Roadmap\n
\n\n
\n Languages and APIs\n
\n
\n In addition to JavaScript, Python, and C/C++, EMS can be added to other languages.\n Languages that can share the EMS API:\n - Scala\n - PHP\n - Java\n - Perl\n - Your name here\n
\n\n\n
\n Examples, Benchmarks, Tests\n
\n
\n Other programs included with the distribution for demonstration, test, and benchmarking purposes.\n\n - Matrix multiply\n - Graph500\n - Sobel Filter\n - MongoDB Replica Server\n
\n\n\n
\n RiverTrail Extensions\n
\n
\n A few years ago a proposed set of language extensions called RiverTrail \n was being supported by Intel and was implemented as a Firefox Plug-In.\n The hooks used by the extension are no longer supported in Firefox, and\n the future of RiverTrail is unclear. To the extent possible, EMS should\n implement RiverTrail extensions.\n
\n EMS internally stores tags that are used for synchronization of user data,\n allowing synchronization to happen independently of the number or kind of processes accessing\n the data.\n The EMS primitives enforce atomic access using automatic state transitions,\n and higher level intrinsics like stacks, queues, and transactional\n memory are built on the EMS primitives.\n\n\n
\n\t
\n\t
\n\t
\n\t
\n\t\t\n\t\t \n\t\t \n\t\t EMS memory is an array of JSON values \n\t\t accessed using atomic operators and/or transactional memory.\n\t\t Safe parallel access is managed by passing through multiple gates:\n\t\t First mapping a key to an index, then \n\t\t accessing user data protected by EMS tags, and completing the whole\n\t\t operation atomically.\n\t\t \n\t\t\n\t
\n\t
\n\t
\n\t
\n\t\t\n\t\t \n\t\t \n\t\t \n\t\t EMS Data Tag Transitions & Atomic operations:\n\t\t F=Full, E=Empty, X=Don't Care, RW=Readers-Writer lock (# of current readers)\n\t\t CAS=Compare-and-Swap, FAA=Fetch-and-Add\n\t\t \n\t\t\n\t
\n\t
\n\t
\n\t
\n
\n \n\n
\n EMS Class & Array Methods\n
\n
\n Operations which inspect or affect the global EMS state are performed\n using class methods of\n the EMS object returned by the require statement.\n Operations that modify the data and tags are performed using methods\n belonging to EMS array to be operated on.\n
Initialize the EMS module, starting\n\t all the other threads. Thread identity and processor affinity\n\t is assigned when the thread is created.\n\t
\n
\n\n
\n\t
ARGUMENTS
\n\t
nThreads
\n\t
<Number>
\n\t
Total number of threads the job should use.\n\t
\n
\n\n
\n\t
\n\t
threadAffinity
\n\t
<Boolean>
\n\t
(Optional, Default = false, Affects only Linux.)\n Set the scheduling affinity of each thread to it's own\n\t core, assigning over-subscribed threads in a round-robin\n\t fashion.\n\t
\n
\n\n
\n\t
\n\t
parallelType
\n\t
<String>
\n\t
(Optional, Default=bsp)\n\t One of bsp, fj, or user. Execution model:\n\t bsp will use EMS' built-in Bulk Synchronous Parallel execution,\n\t fj uses EMS' built-in Fork-join execution\n\t user creates no parallelism\n\t
\n
\n\n
\n\t
\n\t
contextName
\n\t
<String>
\n\t
(Optional, Default=anonymous)\n\t Unique name of parallel context being initialized, required to distinguish\n\t between multiple EMS parallel programs running simultaneously on the\n\t same system.\n\t
\n
\n\n
\n \n
\n
\n\t
RETURNS
\n\t
ems = {\n nThreads : Number, // Number of threads executing\n myID : Number // This thread's ID 0..nThreads-1\n}
\n
\n\n
\n\t
EXAMPLES
\n\t
ems = require('ems')(process.argv[2])
\n\t
Use the first command line argument as the number of nodes:\n\t node foo.js 4 executes using 4 threads.
Attach to an existing or create a new EMS array. \n\t Creation of new (do not use existing) EMS memory regions implies a barrier,\n\t whereas using an existing EMS file will block until the file exists.\n\t
\n
\n\n
\n\t
ARGUMENTS
\n\t
nElements
\n\t
<Number>
\n\t
(Optional, Default is 1) Maximum number of elements in the EMS array or map.
\n
\n
\n\t
\n\t
\n\t
<Array>
\n\t
An array of dimensions of a multi-dimensional array. \n\t A 100×30×50 cube is described by [100, 30, 50].
\n
\n
\n\t
\n\t
heapSize
\n\t
<Number>
\n\t
\n\t (Optional, Default is 0) \n\t Maximum number of bytes reserved for strings,\n\t arrays, maps, and object elements in this array. \n\t The actual amount of memory allocated for use is \n\t rounded up to the nearest power of 2\n\t (until a better memory allocator is implemented).\n\t Additional memory is allocated for bookkeeping.\n\t Memory block size is defined in ems_alloc.h: EMS_MEM_BLOCKSZ\n\t but should be configurable at create time.\n\t
\n
\n
\n\t
\n\t
fileName
\n\t
<String>
\n\t
\n\t (Optional, Default is anonymous) Fully qualified file name\n\t of the file to use as a persistent backing store for the EMS array,\n\t tags, and bookkeeping information.\n\t
\n
\n
\n\t
\n\t
emsArrayDescriptor
\n\t
<Object>
\n\t
\n\t (Alternative to two scalar argument) A complete EMS Array\n\t descriptor may be passed as the only argument instead of\n\t scalar arguments.\nemsArrayDescriptor = {\n dimensions : 100, // Required: Max # elements in EMS array\n // Integer for 1-D, or array of dims [x, y, z]\n heapSize : 100000, // Optional, default=0: Space, in bytes, for \n // strings, maps, and objects. Rounded up to nearest power of two\n mlock : 0, // Optional, default=0%: % of EMS memory to lock into RAM\n useMap : true, // Optional, default=false: Map keys to indexes\n useExisting : true, // Optional, default=false: \n // Preserve data if an file already exists\n persist : true, // Optional, default=true: \n // Preserve the file after threads exit\n doDataFill : false, // Optional, default=false: Initialize memory\n dataFill : undefined, // Optional, If this property is defined, \n // the EMS memory is filled with this value\n setFEtags : 'full', // Optional, If defined, set 'full' or 'empty'\n filename : '/path/to/file' // Optional, default=anonymous: \n // Path to the persistent file of this array\n}\n\t
\n
\n\n\n
\n \n
\n
\n\t
RETURNS
\n\t
<EMS Array>
\n
\n\n
\n\t
EXAMPLES
\n\t
var foo = ems.new(nItems)
\n\t
Create a new non-persistent shared memory EMS\n\t array with no heap space. Scalar data (Number, Boolean, Undefined)\n\t may be stored in this array, but not strings, arrays, or objects.\n\t
\n
\n\n
\n\t
\n\t
var foo = ems.new(nItems, size,\n '/tmp/EMS_foo')
\n\t
Create a file-backed EMS shared memory\n\t space with the filename /tmp/EMS_foo. In addition to the \n\t scalar storage, \n\t space for strings totaling\n\t size bytes is also reserved for strings, arrays, objects, and maps.
\n
\n
\n\t
\n\t
var x = ems.new(nItems, size)
\n\t
Create a new non-persistent shared memory EMS\n\t array that has space for strings totaling size bytes
\n
\n
\n\n\n\n \n\n
Parallel Region (Fork-Join execution only)
\n
\n
\n\t
CLASS METHOD
\n\t
ems.parallel( [args, ...] func )
\n
\n\n
\n\t
SYNOPSIS
\n\t
When using fork-join execution, the master\n\t\tthread enters a parallel region by executing func\n\t\tonce from each process. The master process first starts all the\n\t\tother processes running the function asynchronously, then executes the function\n\t\titself synchronously. All processes join (perform a barrier)\n\t\tafter completion of the work function.\n\t\tThe results of the function are discarded. Global variables on each node\n\t\tare persistent between parallel regions.\n\t
\n
\n\t\t
\n\t\t\t
ARGUMENTS
\n\t\t\t
args
\n\t\t\t
<Any>
\n\t\t\t
Zero or more arguments to be passed to the function.\n\t\t\t
\n\t\t
\n\t\t
\n\t\t\t
\n\t\t\t
func
\n\t\t\t
<Function>
\n\t\t\t
Function to be executed once on every thread. The\n\t\t\t\toptional arguments are used when calling the function. Return value is ignored.\n\t\t\t
\n\t\t
\n
\n \n
\n\t\t
\n\t\t\t
EXAMPLES
\n\t\t\t
ems.parallel( doWork )
\n\t\t\t
The function doWork is executed by every thread.
\n\t\t
\n\t\t
\n\t\t\t
\n\t\t\t
ems.parallel( foo, \"Smith\", 123, doWork )
\n\t\t\t
The function call doWork(foo, \"Smith\", 123)\n\t\t\t\tis performed once by each process.
\n\t\t
\n
\n\n\n \n\n\n
Parallel Loops
\n
\n
\n\t
CLASS METHOD
\n\t
ems.parForEach( first, last, function [, scheduling [, minChunk] ] )
\n
\n\n
\n\t
SYNOPSIS
\n\t
Parallel loop execution, distributing\n\t the iterations among all the threads. The function is invoked\n\t with one argument, the current iteration number. Iterations are\n\t divided among the threads according to the \n\t scheduling method specified.\n\t Parallel for loops must not be nested.\n\t The system should check for this and fall back on serial execution.\n\t A barrier is implied at\n\t the end.
\n\t
\n
\n\n
\n\t
ARGUMENTS
\n\t
first
\n\t
<Number>
\n\t
Index to start iterating
\n
\n
\n\t
\n\t
last
\n\t
<Number>
\n\t
Index to stop iterating (non-inclusive).
\n
\n
\n\t
\n\t
func
\n\t
<Function>
\n\t
Loop body, only input argument is current loop index:\n\t function foo(idx) {...}
\n
\n
\n\t
\n\t
scheduling
\n\t
<String>
\n\t
\n\t guided [minChunk]: Decreasing amounts of work are assigned to each task until minChunk iterations per thread is reached. Load balancing occurs when new chunks are assigned to threads. \n\t static: Equal number of iterations are given to each thread, no dynamic load balancing is performed. \n\t dynamic: All threads share one index which is atomically\n\t incremented by 1 after each iteration. Provides ideal load balancing at the\n\t cost of high per-iteration overhead. \n\t
\n
\n\t
\n\t
minChunk
\n\t
<Number>
\n\t
(Optional, only used when scheduling='guided', default=1) Minimum number of iterations assigned to a single thread.\n\t
\n
\n
\n \n
\n
\n\t
EXAMPLES
\n\t
ems.parForEach(0, nItems-1, func)
\n\t
Execute the func function nItems-1 times with indexes \n\t 0..nItems-1, inclusive.
Distribute iterations numbered 10,000-20,000 (inclusive) using the guided\n\t method with a minimum chunk size of 200 iterations
\n
\n\n
\n\t
\n\t
ems.parForEach(0, 999,\n func, 'static')
\n\t
Execute func() 1,000 times with indexes\n\t 0..999 inclusive. Distribute the iterations evenly across\n\t threads in contiguous blocks.
\n
\n
\n\n\n\n \n\n
Barrier Synchronization
\n
\n
\n\t
CLASS METHOD
\n\t
ems.barrier()
\n
\n\n
\n\t
SYNOPSIS
\n\t
All the threads must reach the same\n\t\tbarrier before proceeding. Failure to call a barrier from every\n\t\tprocess will result in deadlock.\n\t\tIf called outside a parallel region, a barrier has no effect.\n\t
\n
\n
\n \n
\n
\n\t
EXAMPLES
\n\t
ems.barrier()
\n\t
All threads reach the barrier before proceeding.
\n
\n \n
\n\n\n \n\n\n
Critical Region
\n
\n
\n\t
CLASS METHOD
\n\t
ems.critical( func )
\n
\n\n
\n\t
SYNOPSIS
\n\t
Perform function func()\n\t mutually exclusive of other threads. Serializes execution through\n\t all critical regions.\n\t Named regions would be more like OpenMP \n\t
\n
\n\n
\n\t
ARGUMENTS
\n\t
func
\n\t
<Function>
\n\t
Function to perform sequentially.
\n
\n
\n \n
\n
\n\t
EXAMPLES
\n\t
ems.critical( function() {\n // Use a shared resources\n} )
\n\t
A shared resource is accessed sequentially, but in no particular order.
\n
\n
\n\n\n\n\n \n\n\n
Execute on Master Thread Only
\n
\n
\n\t
CLASS METHOD
\n\t
ems.master( func )
\n
\n\n
\n\t
SYNOPSIS
\n\t
Perform function func()\n\t only on thread 0, implies a barrier.
Perform function func()\n\t only once by the first thread to reach the statement. \n\t Implies a barrier.
\n
\n\n
\n\t
ARGUMENTS
\n\t
func
\n\t
<Function>
\n\t
Function to be performed once.
\n
\n
\n \n
\n
\n\t
EXAMPLES
\n\t
ems.single( function() {\n console.log(\"Only first task\")\n} )
\n\t
Console logging is performed only once.
\n
\n
\n\n\n\n\n\n \n\n\n
Print Diagnostic Message
\n\n
\n
\n\t
CLASS METHOD
\n
ems.diag( message )
\n
\n\n
\n\t
SYNOPSIS
\n\t
Print a diagnostic message to the console with a prefix indicating the task ID.
\n
\n\n
\n\t
ARGUMENTS
\n\t
message
\n\t
<String>
\n\t
Text of message to print to the console
\n
\n
\n \n
\n
\n\t
EXAMPLES
\n\t
ems.diag( \"Hello, world!\" ) \n\t
\n\t
EMS 3: Hello, world! appears on console
\n
\n
\n\n\n\n\n\n\n \n \n \n \n \n\n\n
\n EMS Array Methods\n
\n\n
Intrinsic Atomic Operations (AMOs)
\n\n
Read EMS Memory
\n\n
\n
\n\t
ARRAY METHOD
\n\t
emsArray.read( index )
\n
\n
\n\t
\n\t
emsArray.readFE( index )
\n
\n
\n\t
\n\t
emsArray.readFF( index )
\n
\n
\n\t
\n\t
emsArray.readRW( index ), emsArray.releaseRW( index )
\n
\n\n
\n\t
SYNOPSIS
\n\t
The read family of EMS\n\t memory operations return the data stored in an EMS array element.\n\t The value may be any JSON type.\n\t \n\t
\n\t
read
\n\t
Immediately and unconditionally returns the\n\t\t\tstored value, ignoring the tag state.\n\t\t\t Reading uninitialized mapped indexes\n\t\t\t\twill return undefined regardless of the default value\n\t\t\t\tthat would be returned with read__, cas, faa.\n\t\t\t\n\t
\n\t
readFE
\n\t
Blocks until the data element is full, then\n\t atomically reads the value and marks it empty.\n\t
\n\n\t
readFF
\n\t
\n\t Blocks until the data element is full, then\n\t atomically reads leaving it full. This allows safe read access\n\t of data which may be updated\n\t simultaneously. readFF ensures mutual exclusion,\n\t and will block if the data is already under a Readers-Writer\n\t lock.\n\t
\n\n\t
readRW, releaseRW
\n\t
\n\t Blocks until the data element is full or\n\t already under a Readers-Writer lock, then increments the\n\t Readers-Writer reference count. The\n\t function emsArray.releaseRW() decrements the reference\n\t count, restoring the state to Full if no readers remain. \n\t
Read dictionary word at index/key idx in the dictionary array.
\n
\n
\n\t
\n\t
var x = arr.readFE(idx)
\n\t
Block until the element at index i \n\t is full, atomically read the value and mark it empty.
\n
\n
\n\t
\n\t
var x = arr.readFF(idx)
\n\t
Block until the element at index i is full, \n\t atomically read the value and leave it full.
\n
\n
\n\t
\n\t
var x = arr.readRW(idx)
\n\t
Acquire a shared-exclusive readers-writer lock.
\n
\n
\n\n\n\n\n \n\n\n
Write EMS Memory
\n\n
\n
\n\t
ARRAY METHOD
\n\t
emsArray.write( index, value )\n\t
\n
\n
\n\t
\n\t
emsArray.writeXE( index, value )
\n
\n
\n\t
\n\t
emsArray.writeXF( index, value )
\n
\n
\n\t
\n\t
emsArray.writeEF( index, value )
\n
\n\n
\n\t
SYNOPSIS
\n\t
Write a value to an element of an EMS array.\n\t \n\t
\n
write
\n\t
Immediately and unconditionally\n\t writes the value to memory. This operation does not honor or\n\t modify the full/empty tag status.\n\t
\n\n
writeXE
\n\t
Unconditionally and atomically writes the value to the data element and marks the element empty.
\n\n
writeXF
\n\t
Unconditionally and atomically writes the value to the data element and marks the element full.
\n\n
writeEF
\n\t
Blocks until the element is empty, and then atomically writes the value and marks the element full.
\n\t
\n\t
\n
\n\n
\n\t
ARGUMENTS
\n\t
index
\n\t
<Number | String>
\n\t
Index in the EMS array of data to read
\n
\n
\n\t
\n\t
value
\n\t
<Any>
\n\t
Primitive value to store in the array at element numbered index.
\n
\n
\n \n
\n
\n\t
EXAMPLES
\n\t
histogram.write(idx, 0)
\n\t
Initialize the value of histogram[idx] to 0.
\n
\n
\n\t
\n\t
dictionary.write(idx, \"Hello\")
\n\t
Add the string \"Hello\" to the EMS\n\t array dictionary at index idx.
\n
\n
\n\t
\n\t
arr.writeXE(i, undefined)
\n\t
Purge the memory at index i of the EMS array arr.
\n
\n
\n\t
\n\t
arr.writeXF(j, 'Mr. Jones')
\n\t
Unconditionally write the\n\t string 'Mr. Jones'to the EMS array arr\n\t at index j and atomically mark the element full.
\n
\n
\n\t
\n\t
arr.writeEF(2, v)
\n\t
Block until the element at index 2\n\t of arr is empty, atomically write the\n\t value v and mark the memory full.
\n
\n
\n\n\n\n \n\n\n\n
Atomic Fetch and Add
\n\n
\n
\n\t
ARRAY METHOD
\n\t
emsArray.faa( index, value )
\n
\n\n
\n\t
SYNOPSIS
\n\t
\n\t Atomically read the array's JSON primitive element (scalar or string, not array or object),\n\t add the value, and write the new value\n\t back to memory. Return the original contents of the memory.\n\t
\n
\n\n
\n\t
ARGUMENTS
\n\t
index
\n\t
<Integer | String>
\n\t
Index of the element in the EMS array emsArray\n\t to atomically add to.
\n
\n
\n\t
\n\t
value
\n\t
< Number | Boolean | String | Undefined >
\n\t
Value to add to the EMS memory.
\n
\n
\n \n
\n
\n\t
RETURNS
\n\t
< Number | Boolean | String | Undefined >
\n\t
The results are the same type as if \n\t a + b were performed.
\n
\n\n
\n\t
EXAMPLES
\n\t
oldVal = \n statistics.faa( timerIdx, elapsed )\n
\n\t
Return the value in memory before the add operation.
\n
\n
\n\t
\n\t
currentSum =\n arr.faa( index, value ) + value\n
\n\t
Return the current value after the atomic operation has occurred.
\n
\n
\n\n\n\n\n \n\n\n\n
Atomic Compare and Swap
\n\n
\n
\n\t
ARRAY METHOD
\n\t
emsArray.cas( index, oldValue, newValue )
\n
\n\n
\n\t
SYNOPSIS
\n\t
\n\t Atomically read the JSON primitive element (scalar or string, not object or array)\n\t stored at the array's index, \n\t compare the original value to oldValue, and if they are equivalent\n\t store the new value. \n\t The CAS operation succeeded\n\t if the value returned is equivalent \n\t to oldValue.\n\t CAS will block until the EMS memory is marked full. CAS is the\n\t equivalent of atomically performing: \n\t if( arr[idx] == oldValue ) then arr[idx] = newValue\n\t
\n
\n\n
\n\t
ARGUMENTS
\n\t
index
\n\t
<Integer | String>
\n\t
Index into the EMS array to update.\n\t
\n
\n
\n\t
\n\t
oldValue
\n\t
<Number | Boolean | String | Undefined>
\n\t
Value to compare to the value stored in memory.\n\t
\n
\n
\n\t
\n\t
newValue
\n\t
<Any >
\n\t
Value to store if the value in memory is oldValue\n\t
\n
\n
\n \n
\n
\n\t
RETURNS
\n\t
< Number | Boolean | String | Undefined >
\n\t
The value in memory when the compare was performed.
Evaluates as true if the lock stored\n\t at arr[index] was acquired.\n\t
\n
\n
\n\t
\n\t
oldWord = users.cas(1234,\n 'Cooking', 'Eating')
\n\t
Attempt to atomically update user 1234's record from the string \n\t 'Cooking' to the string 'Eating'
\n
\n
\n\n\n\n \n \n\n\n \n\n
Composed Array Operations
\n
\n Composed operations use EMS intrinsics to perform \n deadlock free atomic operations\n involving multiple EMS elements. The composed operations use the tags\n and require data to be full or empty as appropriate for the semantics of the operation.\n
\n\n
Transactional Processing of Multiple Elements
\n\n
\n
\n\t
CLASS METHOD
\n\t
ems.tmStart( tmElements )
\n
\n
\n\t
\n\t
ems.tmEnd( tmHandle, doCommit )
\n
\n\n
\n\t
SYNOPSIS
\n\t
\n\t Lock (by transitioning tags from Full to Empty)\n\t one or more EMS elements in a deadlock\n\t free way. When multiple locks must be acquired, this function\n\t guarantees at least one thread will always make progress. The\n\t optional third element indicates the element is read-only and will \n\t not be modified by the task while the lock is held. Read-only\n\t data is locked using a Readers-Writer lock, permitting additional concurrency.\n\t \n\t Performing transactions within transactions can result in deadlock\n\t if the thread tries to recursively lock an element.\n\t
\n
\n\n
\n\t
ARGUMENTS
\n\t
tmElements
\n\t
<Array>
\n\t
Array identifying which EMS array \n\t elements should be locked. Each array element is itself an array \n\t naming the EMS array and index/key of the data \n\t and an optional Read-Only hint:\n\t [ emsArray, index (, isReadOnly) ]\n\t
\n
\n
\n\t
\n\t
tmHandle
\n\t
<Object>
\n\t
Returned from tmStart(),\n\t contains state information needed to abort or commit the transaction.\n\t
\n
\n
\n\t
\n\t
doCommit
\n\t
<Boolean>
\n\t
Commits the transaction if true,\n\t or aborts and rolls back the transaction if false or undefined.\n\t
Acquire and free locks on the elements in lockList.\n\t Element arrA[idxA0] may be modified, but elements \n\t arrA[idxA1] and arrB[idxB0] are read-only.\n\t
\n
\n
\n\t
\n\t
ems.tmEnd( tm, true )\n\t
\n\t
Commit the transaction\n\t
\n
\n
\n\n\n\n\n\n\n \n\n\n\n
Stacks & Queues
\n\n
\n
\n\t
ARRAY METHOD
\n\t
emsArray.push( value )
\n
\n
\n\t
\n\t
emsArray.pop( )
\n
\n
\n\t
\n\t
emsArray.enqueue( value )
\n
\n
\n\t
\n\t
emsArray.dequeue( )
\n
\n\n
\n\t
SYNOPSIS
\n\t
Append or remove data from a LIFO or\n\t FIFO. If the queue or stack is empty, the pop\n\t or dequeue operation returns\n\t Undefined, which is indistinguishable from an Undefined\n\t that was explicitly pushed onto the stack.\n\t
\n
\n\n
\n\t
ARGUMENTS
\n\t
value
\n\t
<Any>
\n\t
\n\t Value to add to the queue or stack.\n\t
\n
\n
\n \n
\n
\n\t
RETURNS
\n\t
emsArray.pop(), emsArray.dequeue() : < Any >
\n
\n
\n\t
\n\t
emsArray.push() : < Number >\nemsArray.enqueue() : < Number >
\n\t
The number of elements presently on the stack or queue
\n
\n\n
\n\t
EXAMPLES
\n\t
comments.push( \"Hello, world\" )
\n\t
Append the string to the EMS array comments
\n
\n
\n\t
\n\t
mostRecent = comments.pop()
\n\t
Atomically return the value\n\t at the top of the stack.
\n
\n
\n\n\n\n\t\n\n\t
Look up a key used to map a value
\n\t
\n\t\t
\n\t\t\t
CLASS METHOD
\n\t\t\t
emsArray.index2key( index )
\n\t\t
\n\n\t\t
\n\t\t\t
SYNOPSIS
\n\t\t\t
\n\t\t\t\tConvert an index into an EMS array to the key used to map\n\t\t\t\ta value to that hashed index. This function can be used\n\t\t\t\tto iterate over all the elements of a mapped array.\n\t\t\t\t
\n\t\t
\n\n\t\t
\n\t\t\t
ARGUMENTS
\n\t\t\t
index
\n\t\t\t
<Number>
\n\t\t\t
\n\t\t\t\tIndex of the element in the EMS array to get the key for
\n\t\t
\n\t
\n\t \n\t
\n\t\t
\n\t\t\t
RETURNS
\n\t\t\t
< Any >
\n\t\t\t
The key used to map the value which hashed to this index.
\n\t\t
\n\t
\n\n\n\t\n\n\t
Freeing EMS Arrays
\n\t
\n\t\t
\n\t\t\t
CLASS METHOD
\n\t\t\t
emsArray.destroy( remove_file )
\n\t\t
\n\n\t\t
\n\t\t\t
SYNOPSIS
\n\t\t\t
\n\t\t\t\tRelease the persistent and non-persistent resources associated with an EMS array,\n\t\t\t\talternatively persisting the data if remove_file is false.\n\t\t\t\tImplies a barrier.\n\t\t\t\t
\n\t\t
\n\n\t\t
\n\t\t\t
ARGUMENTS
\n\t\t\t
remove_file
\n\t\t\t
<Boolean>
\n\t\t\t
\n\t\t\t\tIf true, remove the file, else allow the EMS file to persist.
\n\t\t
\n\t
\n\t \n\t
\n\t\t
\n\t\t\t
RETURNS
\n\t\t\t
\n\t\t\t
None.
\n\t\t
\n\t
\n\n\n\t\n\n\t
Synchronize EMS Memory to Persistent Storage
\n\t
\n\t\t
\n\t\t\t
CLASS METHOD
\n\t\t\t
emsArray.sync( [ index [, nElements] ] )
\n\t\t
\n\n\t\t
\n\t\t\t
SYNOPSIS
\n\t\t\t
\n\t\t\t\tSynchronize the EMS memory with persistent storage.\n\t\t\t\t
\n\t\t
\n\n\t\t
\n\t\t\t
ARGUMENTS
\n\t\t\t
index
\n\t\t\t
<String | Number>
\n\t\t\t
\n\t\t\t\t(Optional, default = entire EMS array)\n\t\t\t\tIndex of the element in the EMS array to synchronize to disk
\n\t\t
\n\t\t
\n\t\t\t
\n\t\t\t
nElements
\n\t\t\t
<Number>
\n\t\t\t
\n\t\t\t\t(Optional, only defined if index is also defined,\n\t\t\t\tdefault = 1)\n\t\t\t\tNumber of sequential indexes, starting with index,\n\t\t\t\tthat should be synchronized to disk
\n\t\t
\n\n\t
\n\t \n\t
\n\t\t
\n\t\t\t
RETURNS
\n\t\t\t
< Boolean >
\n\t\t\t
True if memory was successfully synchronized to disk,\n\t\t\t\totherwise false.
\n\t\t
\n\n\t\t
\n\t\t\t
EXAMPLES
\n\t\t\t
users.sync( userID )
\n\t\t\t
The user's record is committed to disk before the function returns.
\n\t\t
\n\t
\n\n\n\t\n\n\n
TODO Reduce
\n\n
\n
\n\t
ARRAY METHOD
\n\t
emsArray.reduce( func )
\n
\n\n
\n\t
SYNOPSIS
\n\t
Perform a parallel reduction on the elements of an EMS array.
\n
\n\n
\n\t
ARGUMENTS
\n\t
func
\n\t
<Function>
\n\t
Function to combine this element with the\n\t partial reduction. Function arguments are func(element,\n\t sum), and the function returns a new sum.\n\t The arguments and results may be of any type.
\n
\n
\n \n
\n
\n\t
RETURNS
\n\t
< Any >
\n
\n\n
\n\t
EXAMPLES
\n\t
sum = distances.reduce(\n function( val, sum ) {\n return(sum+val) } )
\n\t
Perform the arithmetic sum on the values in the array.
Reorder the array indexes to the\n\t ordering specified in the\n\t array order. If order contains the same\n\t index more than once, the output for those indexes is\n\t undefined.
\n
\n\n
\n\t
ARGUMENTS
\n\t
order
\n\t
<Array>
\n\t
Integer permutation array.
\n
\n
\n \n
\n
\n\t
EXAMPLES
\n\t
arr.permute( order )
\n\t
Reorder the array arr\n\t to the new element ordering.
\n
\n
\n\n\n\n \n\n\n\n
TODO: Scan
\n\n
\n
\n\t
ARRAY METHOD
\n\t
emsArray.scan( func )
\n
\n\n
\n\t
SYNOPSIS
\n\t
Perform a parallel prefix operation on\n\t the EMS array elements: \n\t a[i] = func(a[i], a[i-1])
\n
\n\n
\n\t
ARGUMENTS
\n\t
func
\n\t
<Function>
\n\t
Function to combine this element and the\n\t previous element. Function arguments are \n\t func(thisElement, previousElement), \n\t and returns a new partial result.
\n
\n
\n \n
\n
\n\t
RETURNS
\n\t
< Array >
\n
\n\n
\n\t
EXAMPLES
\n\t
partialSums = subtotals.scan(\n function(val, sum ) { return(sum+val) } )
\n\t
Return a EMS array with the partial sums in each element.
\n
\n
\n\n\n\n\n \n\n\n\n
TODO: Map
\n\n
\n
\n\t
ARRAY METHOD
\n\t
emsArray.map( func [, name] )
\n
\n\n
\n\t
SYNOPSIS
\n\t
Produce a new EMS array of the same length\n\t by invoking func()\n\t for each element in the array to produce the corresponding element\n\t in the new EMS array.\n\t
\n
\n\n
\n\t
ARGUMENTS
\n\t
func
\n\t
<Function>
\n\t
Function to perform on each array element. The function is invoked with one argument, the EMS primitive element, and returns a new primitive element.
\n
\n
\n\t
\n\t
Persistent filename
\n\t
<String>
\n\t
(Optional, default is non-persistent) \n\t If the new mapping should have a persistent backing on a filesystem, \n\t this assigns the fully qualified filename.
Scales each element in the array values from inches to cm, producing a new EMS array with scaled values.
\n
\n
\n\n\n\n \n\n\n\n
TODO: Filter
\n\n
\n
\n\t
ARRAY METHOD
\n\t
emsArray.filter( func [, name] )
\n
\n\n
\n\t
SYNOPSIS
\n\t
Produce a new EMS array containing only the elements\n\t which evaluate as true when passed as the argument to func(element).\n\t The elements of the new array may appear in any order.\n\t
\n
\n\n
\n\t
ARGUMENTS
\n\t
func
\n\t
<Function>
\n\t
Function to perform on each array element. The function is invoked with one argument, the EMS primitive element, and returns a boolean indicating if the element should be included in the new array.
\n
\n
\n\t
\n\t
name
\n\t
<String>
\n\t
(Optional, default is non-persistent) \n\t If the new mapping requires persistent backing on a filesystem, \n\t the fully qualified filename is defined.
\n\n\n\n \n\n"
},
{
"path": "Examples/Interlanguage/README.md",
"content": "# Inter-Language Persistent Shared Memory\n\nIn *The Future™*, memory will have new capabilities that\nprogrammers will make decisions about how best to use:\n- __Non-Volatility__ - No more \"loading\" files into variables in memory,\n just use the same variables as last time.\n- __Sharing__ - Directly sharing objects between programs instead of\n communicating them through network messages.\n- __Selectable Latency__ - Expensive fast memory for frequently accessed variables,\n cheaper slower memory for less frequently used data.\n- __Correctness__ - Very inexpensive bulk memory might give approximate values instead\n of the exact value stored.\n- __Location__ - Allocate memory in another another rack or data center\n- __Mirroring__ - Automatic fail-over to backup memory \n \nEMS presently supports the first two and is ready\nto incorporate new memory capabilities as they become available.\n\n## Sharing Objects Between Programs\n\nSharing EMS data between programs written in different languages \nis no different than sharing data between programs in the same language,\nor sharing data between same program run multiple times.\n\n\n\n### Javascript Semantics in Other Languages\n\nEvery language has it's own semantics and storage sequences for it's data structures,\nEMS makes the various representations interoperable using Javascript's semantics.\nThat is, if your Python program relies on horrifying semantics like:\n\n```python\n>>> a = [1,2,3]\n>>> a + \"abc\"\nTraceback (most recent call last):\n File \"\", line 1, in \nTypeError: can only concatenate list (not \"str\") to list\n>>> a += \"abc\"\n>>> a\n[1, 2, 3, 'a', 'b', 'c']\n>>>\n``` \n\nYou will be horrified to learn EMS will instead follow Javascript's horrifying semantics:\n\n```javascript\n> a = [1,2,3]\n[ 1, 2, 3 ]\n> a + 'abc'\n'1,2,3abc'\n> a += 'abc'\n'1,2,3abc'\n>\n```\n\nYou may be nonplussed to learn this is *The Future™*.\nI, too, was horrified and baffled at this discovery. \nAlas, this is an inevitable truth of *The Future™*.\n\n\n### Possible Orders for Starting Processes\n\nTwo processes sharing memory may start in any order, meaning programs\nmust take care when initializing to not overwrite another program which\nmay also be trying to initialize, or detect when it is necessary to\nrecover from a previous error.\n\n| Persistent EMS Array File | (Re-)Initialize | Use |\n| :------------- |:-------------:| :-----:|\n| Already exists | A one-time initialization process truncates and creates the EMS array. The first program to start must create the EMS file with the `useExisting : false` attribute | Any number of processes may attach to the EMS array in any order using the attribute `useExisting : true` |\n| Does not yet exist | Subsequent programs attach to the new EMS file with the `useExisting : true` attribute | N/A |\n\n\n# Running the Example\nThe programs `interlanguage.py` and `interlanguage.js` are complementary\nto each other, and must be started in the correct order:\n\n__First start the Javascript program, then start the Python.__\n\n```bash\ndunlin> node interlanguage.js\nJS Started, waiting for Python to start...\n [ EXECUTION STALLS UNTIL PYTHON PROGRAM STARTS ]\nHello from Python\n...\n``` \n\nSecond, start the Python program\n```bash\ndunlin> python interlanguage.py\nStart this program after the JS program.\nIf this Python appears hung and does not respond to ^C, also try ^\\ and ^Z\n----------------------------------------------------------------------------\nHello from Javascript\n...\n```\nDue to a well-intentioned signal mask in the Python runtime meant to\nprovide resiliency in the face of a failing native plug-in module,\nsometimes Python will appear hung when it is blocked on an EMS operation.\nIn these situations an alternative way to kill the Python interpreter \nis with SIGQUIT (Control-\\ from most shells), or back-grounding the\njob with Control-Z and sending the job a kill signal. \n\n \n### What the Example is Doing\n- Each process writes a hello message to an EMS variable, marking it full.\n- Each process reads the variable written by the other process, blocking until it has been written\n- A simple point-to-point barrier is demonstrated by handshaking\n- Illustrate some of the differences between Python and Javascript nested objects\n- Both JS and Py enter a loop that increments a shared counter\n- JS checks the results to make sure every iteration occurred once\n- Illustrate some quirks of language design in JS and Py\n- JS leaves a shared counter running in the background via `setInterval`\n- Python enters a loop that occasionally reads the shared counter, showing it being updated\n- Python over-writes the counter with `stop`\n- JS notices the counter has been replaced with `stop` and exits.\n\n\n\n### Interactive Example\nIllustration of sharing data between the Node.js and Python \ndone interactively from the REPLs.\n \n\n"
},
{
"path": "Examples/Interlanguage/interlanguage.js",
"content": "/*-----------------------------------------------------------------------------+\n | Extended Memory Semantics (EMS) Version 1.4.5 |\n | Synthetic Semantics http://www.synsem.com/ mogill@synsem.com |\n +-----------------------------------------------------------------------------+\n | Copyright (c) 2017, Jace A Mogill. All rights reserved. |\n | |\n | Redistribution and use in source and binary forms, with or without |\n | modification, are permitted provided that the following conditions are met: |\n | * Redistributions of source code must retain the above copyright |\n | notice, this list of conditions and the following disclaimer. |\n | * Redistributions in binary form must reproduce the above copyright |\n | notice, this list of conditions and the following disclaimer in the |\n | documentation and/or other materials provided with the distribution. |\n | * Neither the name of the Synthetic Semantics nor the names of its |\n | contributors may be used to endorse or promote products derived |\n | from this software without specific prior written permission. |\n | |\n | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS |\n | \"AS IS\" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT |\n | LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR |\n | A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL SYNTHETIC |\n | SEMANTICS LLC BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, |\n | EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, |\n | PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR |\n | PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF |\n | LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING |\n | NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS |\n | SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. |\n | |\n +-----------------------------------------------------------------------------*/\n/*\n Start this JS program first, then start the Python program\n\n Possible Orders for Starting Processes\n ---------------------------------------------\n\n A persistent EMS file already exists\n 1. A one-time initialization process creates the EMS array\n 2. Any number of processes my attach to the EMS array in\n any order.\n\n\n A new EMS array will be created\n 1. The first program to start must create the EMS file with\n the \"useExisting : false\" attribute\n 2. Subsequent programs attach to the new EMS file with the\n \"useExisting : true\" attribute\n\n */\n\"use strict\";\n// Initialize EMS with 1 process, no thread-CPU affinity,\n// \"user\" mode parallelism, and a unique namespace for EMS runtime\n// (\"jsExample\") to keep the JS program distinct from Python EMS\n// program also running.\nlet ems = require(\"ems\")(1, false, \"user\", \"jsExample\");\n\nif (Number(process.version[1]) < 6) {\n console.log('Requires Node.js version 6.x or higher -- will not work with polyfill Harmony');\n process.exit(1);\n}\n\n\nconst maxNKeys = 100;\nconst bytesPerEntry = 100; // Bytes of storage per key, used for key (dictionary word) itself\nlet shared = ems.new({\n \"ES6proxies\": true, // Enable native JS object-like syntax\n \"useExisting\": false, // Create a new EMS memory area, do not use existing one if present\n \"filename\": \"/tmp/interlanguage.ems\", // Persistent EMS array's filename\n \"dimensions\": maxNKeys, // Maximum # of different keys the array can store\n \"heapSize\": maxNKeys * bytesPerEntry,\n \"setFEtags\": \"empty\", // Set default full/empty state of EMS memory to empty\n \"doSetFEtags\": true,\n \"useMap\": true // Use a key-index mapping, not integer indexes\n});\n\n\n\n//------------------------------------------------------------------------------------------\n// Begin Main Program\nconsole.log(\"JS Started, waiting for Python to start...\");\n// One-time initialization should be performed before syncing to Py\nshared.writeXF(\"nestedObj\", undefined);\n\n// Initial synchronization with Python\n// Write a value to empty memory and mark it full\nshared.writeEF(\"JS hello\", \"from Javascript\");\n\n// Wait for Python to start\nconsole.log(\"Hello \" + shared.readFE(\"Py hello\"));\n\n/*\n This synchronous exchange was a \"barrier\" between the two processes.\n\n The idiom of exchanging messages by writeEF and readFE constitutes\n a synchronization point between two processes. A barrier synchronizing\n N tasks would require N² variables, which is a reasonable way to\n implement a barrier when N is small. For larger numbers of processes\n barriers can be implemented using shared counters and the Fetch-And-Add\n (FAA) EMS instruction.\n\n The initialization of EMS values as \"empty\" occurs when the EMS array\n was created with:\n \"setFEtags\": \"empty\",\n \"doSetFEtags\": true,\n If it becomes necessary to reset a barrier, writeXE() will\n unconditionally and immediately write the value and mark it empty.\n */\n\n\n// Convenience function to synchronize with another process\n// The Python side reverses the barrier names\nfunction barrier(message) {\n console.log(\"Entering Barrier:\", message);\n shared.writeEF(\"py side barrier\", undefined);\n shared.readFE(\"js side barrier\");\n console.log(\"Completed Barrier.\");\n console.log(\"---------------------------------------------------\");\n}\n\n\n// --------------------------------------------------------------------\nbarrier(\"Trying out the barrier utility function\");\n// --------------------------------------------------------------------\n\n\n// JS and Python EMS can read sub-objects with the same syntax as native objects,\n// but writes are limited to only top level attributes. This corresponds to the\n// implied \"emsArray.read(key).subobj\" performed\nshared.top = {};\nconsole.log(\"Assignment to top-level attributes works normally:\", shared.top);\nshared.top.subobj = 1; // Does not take effect like top-level attributes\nconsole.log(\"But sub-object attributes do not take effect. No foo?\", shared.top.subobj);\n\n// A nested object can be written at the top level.\nshared.nestedObj = {\"one\":1, \"subobj\":{\"left\":\"l\", \"right\":\"r\"}};\nconsole.log(\"Nested object read references work normally\", shared.nestedObj.subobj.left);\n\n// A workaround to operating on nested objects\nlet nestedObjTemp = shared.nestedObj;\nnestedObjTemp.subobj.middle = \"m\";\nshared.nestedObj = nestedObjTemp;\n\n// The explicitly parallel-safe way to modify objects is similar to that \"workaround\"\nnestedObjTemp = shared.readFE(\"nestedObj\");\nnestedObjTemp.subobj.front = \"f\";\nshared.writeEF(\"nestedObj\", nestedObjTemp);\n\n\n// --------------------------------------------------------------------\nbarrier(\"This barrier matches where Python is waiting to use nestedObj\");\n// --------------------------------------------------------------------\n\n// Python does some work now\n// Initialize the counter for the next section\nshared.writeXF(\"counter\", 0);\n\n// --------------------------------------------------------------------\nbarrier(\"JS and Py are synchronized at the end of working with nestedObj\");\n// --------------------------------------------------------------------\n\n// ---------------------------------------------------------------------------\n// Use the EMS atomic operation intrinsics to coordinate access to data\n// This is 10x as many iterations as Python in the same amount of time\n// because V8's TurboFan compiler kicks in.\nfor (let count = 0; count < 1000000; count += 1) {\n let value = shared.faa(\"counter\", 1);\n if (count % 100000 == 0) {\n console.log(\"JS iteration\", count, \" Shared Counter=\", value)\n }\n}\n\n\n// --------------------------------------------------------------------\nbarrier(\"Waiting for Py to finish it's counter loop\");\n// --------------------------------------------------------------------\n\nconsole.log(\"The shared counter should be 11000000 ==\", shared.counter);\n\n\n\n// ---------------------------------------------------------------------------\n// Ready to earn your Pro Card?\n\n// Wait for Py to initialize the array+string\nbarrier(\"Wait until it's time to glimpse into the future of Javascript\");\nconsole.log(\"The array+string from Py:\", shared.arrayPlusString);\n\nbarrier(\"Waiting for shared.arrayPlusString to be initialized\");\nconsole.log(\"JS can do array+string, it produces a string:\", shared.arrayPlusString + \" world!\");\nshared.arrayPlusString += \" RMW world!\";\nconsole.log(\"JS performing \\\"array + string\\\" produces a self-consistent string:\", shared.arrayPlusString);\nbarrier(\"Arrive at the future.\");\n\n\n// ---------------------------------------------------------------------------\n// Leave a counter running on a timer\n// First re-initialize the counter to 0\nshared.writeXF(\"counter\", 0);\n\nfunction incrementCounter() {\n // Atomically increment the shared counter\n let oldValue = shared.faa(\"counter\", 1);\n\n // If the counter has been set to \"Stop\", clear the timer interval\n if (oldValue == \"stop\") {\n console.log(\"Timer counter was signaled to stop\");\n clearInterval(timerCounter);\n }\n\n // Show counter periodically\n if (oldValue % 100 == 0) {\n console.log(\"Counter =\", shared.counter);\n }\n}\n\nvar timerCounter = setInterval(incrementCounter, 10); // Increment the counter every 1ms \n\nbarrier(\"Welcome to The Future™!\");\n// Fall into Node.js event loop with Interval Timer running\n"
},
{
"path": "Examples/Interlanguage/interlanguage.py",
"content": "#!/usr/bin/env python\n# -*- coding: utf-8 -*-\n\"\"\"\n +-----------------------------------------------------------------------------+\n | Extended Memory Semantics (EMS) Version 1.4.5 |\n | Synthetic Semantics http://www.synsem.com/ mogill@synsem.com |\n +-----------------------------------------------------------------------------+\n | Copyright (c) 2017, Jace A Mogill. All rights reserved. |\n | |\n | Redistribution and use in source and binary forms, with or without |\n | modification, are permitted provided that the following conditions are met: |\n | * Redistributions of source code must retain the above copyright |\n | notice, this list of conditions and the following disclaimer. |\n | * Redistributions in binary form must reproduce the above copyright |\n | notice, this list of conditions and the following disclaimer in the |\n | documentation and/or other materials provided with the distribution. |\n | * Neither the name of the Synthetic Semantics nor the names of its |\n | contributors may be used to endorse or promote products derived |\n | from this software without specific prior written permission. |\n | |\n | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS |\n | \"AS IS\" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT |\n | LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR |\n | A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL SYNTHETIC |\n | SEMANTICS LLC BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, |\n | EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, |\n | PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR |\n | PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF |\n | LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING |\n | NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS |\n | SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. |\n | |\n +-----------------------------------------------------------------------------+\n\"\"\"\n\"\"\"\n tl;dr Start the Javascript first, then the Python\n --------------------------------------------------\n\n See interlanguage.js for more detailed description of starting processes\n sharing EMS memory\n\n NOTE ABOUT \"STUCK\" JOBS\":\n Python runtime can get \"stuck\" and may not respond to Control-C\n but can still be interrupted with Control-Z or Control-\\\n This \"stuck\" behavior is due to how the Python/C interface protects\n the Python runtime.\n\"\"\"\n\nimport time\nimport sys\nimport random\nsys.path.append(\"../../Python/\")\nimport ems\n\n# Initialize EMS: 1 process, no thread-core affinity, user provided parallelism\n# Initialize EMS with 1 process, no thread-CPU affinity,\n# \"user\" mode parallelism, and a unique namespace for EMS runtime\n# (\"pyExample\") to keep the JS program distinct from Javascript EMS\n# program also running.\nems.initialize(1, False, \"user\", \"pyExample\")\n\n\n# The EMS array attributes must be the same by all programs sharing the array,\n# with the exception of the the initial file creation which uses \"useExisting:False\"\n# instead of \"useExisting:True\"\nmaxNKeys = 100\nbytesPerEntry = 100 # Bytes of storage per key, used for key (dictionary word) itself\nshared = ems.new({\n 'useExisting': True, # Connect the EMS memory created by JS, do not create a new memory\n 'filename': \"/tmp/interlanguage.ems\", # Persistent EMS array's filename\n 'dimensions': maxNKeys, # Maximum # of different keys the array can store\n 'heapSize': maxNKeys * 100, # 100 bytes of storage per key, used for key (dictionary word) itself\n 'useMap': True # Use a key-index mapping, not integer indexes\n})\n\n\n# ------------------------------------------------------------------------------------------\n# Begin Main Program\nprint \"\"\"Start this program after the JS program.\nIf this Python appears hung and does not respond to ^C, also try ^\\ and ^Z\n----------------------------------------------------------------------------\n\"\"\"\n\n# Initial synchronization with JS\n# Write a value to empty memory and mark it full\nshared.writeEF(\"Py hello\", \"from Python\")\n\n# Wait for JS to start\nprint \"Hello \", shared.readFE(\"JS hello\")\n\n\ndef barrier(message):\n \"\"\"Define a convenience function to synchronize with another process\n The JS side reverses the barrier names\"\"\"\n global shared\n naptime = random.random() + 0.5\n time.sleep(naptime) # Delay 0.5-1.5sec to make synchronization apparent\n print \"Entering Barrier:\", message\n shared.writeEF(\"js side barrier\", None)\n shared.readFE(\"py side barrier\")\n print \"Completed Barrier after delaying for\", naptime, \"seconds\"\n print \"------------------------------------------------------------------\"\n\n\nprint \"Trying out the new barrier by first napping for 1 second...\"\ntime.sleep(1)\n# ---------------------------------------------------------------------\nbarrier(\"Trying out the barrier utility function\")\n# ---------------------------------------------------------------------\n\n\n# --------------------------------------------------------------------\n# This barrier matches where JS has set up \"shared.top\", and is now waiting\n# for Python to finish at the next barrier\nbarrier(\"Finished waiting for JS to finish initializing nestedObj\")\n\nprint \"Value of shared.nestedObj left by JS:\", shared.nestedObj\ntry:\n shared.nestedObj.number = 987 # Like JS, setting attributes of sub-objects will not work\n print \"ERROR -- This should not work\"\n exit()\nexcept:\n print \"Setting attributes of sub-objects via native syntax will not work\"\n\nprint \"These two expressions are the same:\", \\\n shared[\"nestedObj\"].read()[\"subobj\"][\"middle\"], \"or\", \\\n shared.read(\"nestedObj\")[\"subobj\"][\"middle\"]\n\n\n# --------------------------------------------------------------------\nbarrier(\"JS and Py are synchronized at the end of working with nestedObj\")\n# --------------------------------------------------------------------\n\n\n# Enter the shared counter loop\nfor count in xrange(100000):\n value = shared.faa(\"counter\", 1)\n if count % 10000 == 0:\n print \"Py iteration\", count, \" Shared counter=\", value\nbarrier(\"Waiting for Js to finish it's counter loop\")\n\n\n# --------------------------------------------------------------------\n# Ready to earn your Pro Card?\n\n# Make JS wait while Python demonstrates a \"create approach to language design\"\nshared.arrayPlusString = [1, 2, 3] # Initialize arrayPlusString before starting next phase\nbarrier(\"Wait until it's time to glimpse into the future of Python\")\ntry:\n dummy = shared.arrayPlusString + \"hello\"\n print \"ERROR -- Should result in: TypeError: can only concatenate list (not \\\"str\\\") to list\"\n exit()\nexcept:\n print \"Adding an array and string is an error in Python\"\n\nshared.arrayPlusString += \"hello\"\nprint \"However 'array += string' produces an array\", shared.arrayPlusString\n\n# Let JS know 'shared.arrayPlusString' was initialized\nbarrier(\"shared.arrayPlusString is now initialized\")\n# JS does it's work now, then wait for JS to do it's work\nbarrier(\"Arrive at the future.\")\n# JS does all the work in this section\nbarrier(\"Welcome to The Future™!\")\n\n\n# --------------------------------------------------------------------\n# JS is now in it's event loop with the counter running\nfor sampleN in xrange(10):\n time.sleep(random.random())\n print \"Shared counter is now\", shared.counter\n\n# Restart the counter \nshared.counter = 0\nfor sampleN in xrange(5):\n time.sleep(random.random() + 0.5)\n print \"Shared reset counter is now\", shared.counter\n\n\n# Wait 1 second and stop the JS timer counter\ntime.sleep(1)\nshared.counter = \"stop\"\n\n# Show the counter has stopped. Last value = \"stop\" + 1\nfor sampleN in xrange(5):\n time.sleep(random.random() + 0.5)\n print \"Shared reset counter should have stopped changing:\", shared.counter\n\nprint \"Exiting normally.\"\n"
},
{
"path": "Examples/KeyValueStore/README.md",
"content": "# Key-Value Store Example\n\nThis example implements a simple REST server, the API is a 1-to-1 \nmapping of REST commands to EMS commands, \neffectively making the EMS shared memory model\nan object store service over the network.\nThe RESTful API inherits all the atomicity\nand persistence properties of the underlying EMS array.\n\n## Parallelism & Deadlocks\n\nThe HTTP web interface used is Node's built-in HTTP module which\nimplements a serial event loop to process HTTP requests one at\ntime in a synchronous (blocking) fashion.\nThe execution time of EMS operations is similar to any other variable reference,\nmaking EMS most efficient as a synchronous programming model,\nhowever, because a `readFE` request may block the Node event loop\nclients susceptible to deadlock.\nA `writeEF` request that would unblock execution cannot handled \nby the single-threaded Node HTTP server that is blocked on a\n`readFE`, effectively deadlocking the system.\n\nThe two potential solutions, asynchronous execution and multi-processing, \nare complementary forms of parallelism that may be used\nto address the deadlock problem:\n\n- __Asynchronous Execution__: \n Blocking calls are turned into continuations that will be scheduled\n for execution again later.\n The number of in-flight operations before potential deadlock is equal\n to the number of continuations the runtime implements.\n \n This is not presently implemented, and would require the addition\n of new non-blocking variants of EMS functions, such as \n `nbReadFE()` and `nbWriteEF()`.\n- __Multi-Processing__: \n Events are distributed across multiple single-threaded event loops.\n Idle threads are assigned new events as they arrive, and each event\n loop completes an event before beginning a new one. The number of\n in-flight operations before potential deadlock is equal to the number\n of processes.\n \n This is implemented using Node's built-in [Cluster](https://nodejs.org/api/cluster.html) module.\n \n\n## Starting the EMS Server process\n\nUsing Node's built-in `cluster` module that forks multiple Node.js\nprocesses, we implement the multiple event loops needed for multi-processing.\nNode's built-in `http` module provides the serial HTTP event loop which is\n then parallelized using the `cluster` module.\n `cluster` manages all child process creation, destruction, and communication.\n \n Starting the web server from the command line:\n ```bash\nnode kv_store.js [# processes]\n```\n\nIf the number of processes is not specified, 8 are started.\n\n## Manually Sending Commands using CURL\n\nAn EMS array is presented as a Key-Value store with EMS memory operations\npresented as a REST interface.\nAn ordinary Node.js Cluster is started to demonstrate parallelism outside of EMS.\n\n```bash\nnode kv_store.js 8 # Start 8 servers, logs will appear on this terminal\n```\n\n```bash\ndunlin> curl localhost:11080/writeXF?foo=Hello # Unconditionally write \"Hello\" to the key \"foo\" and mark Full\ndunlin> curl localhost:11080/faa?foo=+World! # Fetch And Add returns original value: \"Hello\"\n\"Hello\"\ndunlin> curl localhost:11080/readFF?foo # Read Full and leave Full, final value is \"Hello World!\"\n\"Hello World!\"\ndunlin>\n```\n\n\n## Producer-Consumer\n\n
\n
\n
\n\n```bash\ndunlin> curl localhost:11080/readFE?bar # EMS memory initialized as empty, this blocks\n```\n
\n
\n\n```bash\ndunlin> curl localhost:11080/readFF?foo # Read Full and leave Full, final value is \"Hello World!\"\n```\n
\n
\n
\n"
},
{
"path": "Examples/KeyValueStore/kv_store.js",
"content": "/*-----------------------------------------------------------------------------+\n | Extended Memory Semantics (EMS) Version 1.4.5 |\n | Synthetic Semantics http://www.synsem.com/ mogill@synsem.com |\n +-----------------------------------------------------------------------------+\n | Copyright (c) 2011-2014, Synthetic Semantics LLC. All rights reserved. |\n | Copyright (c) 2015-2017, Jace A Mogill. All rights reserved. |\n | |\n | Redistribution and use in source and binary forms, with or without |\n | modification, are permitted provided that the following conditions are met: |\n | * Redistributions of source code must retain the above copyright |\n | notice, this list of conditions and the following disclaimer. |\n | * Redistributions in binary form must reproduce the above copyright |\n | notice, this list of conditions and the following disclaimer in the |\n | documentation and/or other materials provided with the distribution. |\n | * Neither the name of the Synthetic Semantics nor the names of its |\n | contributors may be used to endorse or promote products derived |\n | from this software without specific prior written permission. |\n | |\n | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS |\n | \"AS IS\" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT |\n | LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR |\n | A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL SYNTHETIC |\n | SEMANTICS LLC BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, |\n | EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, |\n | PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR |\n | PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF |\n | LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING |\n | NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS |\n | SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. |\n | |\n +-----------------------------------------------------------------------------*/\n'use strict';\nvar ems = require('ems')(1, false, 'user');\nvar http = require('http');\nvar url_module = require(\"url\");\nvar cluster = require('cluster');\nvar port = 8080;\nvar shared_counters;\nvar persistent_data;\nvar myID;\nvar persistent_data_options = {\n dimensions: [100000],\n heapSize: [10000000],\n useExisting: false,\n useMap: true,\n filename: '/tmp/persistent_shared_web_data.ems'\n};\n\nvar shared_counters_options = {\n dimensions: [100],\n heapSize: [10000],\n useExisting: false,\n useMap: true,\n filename: '/tmp/counters.ems'\n};\n\n\nfunction handleRequest(request, response) {\n var requestN = shared_counters.faa('nRequests', 1); // Increment the count of requests\n var parsed_request = url_module.parse(request.url, true);\n var key = Object.keys(parsed_request.query)[0];\n var value = parsed_request.query[key];\n if (value === '') { value = undefined; }\n var data;\n var opname;\n\n if (!key) {\n response.end(\"ERROR: No key specified in request\");\n return;\n }\n\n if (parsed_request.pathname.indexOf(\"/readFE\") >= 0) {\n data = persistent_data.readFE(key);\n opname = 'readFE';\n } else if (parsed_request.pathname.indexOf(\"/readFF\") >= 0) {\n data = persistent_data.readFF(key);\n opname = 'readFF';\n } else if (parsed_request.pathname.indexOf(\"/read\") >= 0) {\n data = persistent_data.read(key);\n opname = 'read';\n } else if (parsed_request.pathname.indexOf(\"/writeXE\") >= 0) {\n data = persistent_data.writeXE(key, value);\n opname = 'writeXE';\n } else if (parsed_request.pathname.indexOf(\"/writeXF\") >= 0) {\n data = persistent_data.writeXF(key, value);\n opname = 'writeXF';\n } else if (parsed_request.pathname.indexOf(\"/writeEF\") >= 0) {\n data = persistent_data.writeEF(key, value);\n opname = 'writeEF';\n } else if (parsed_request.pathname.indexOf(\"/write\") >= 0) {\n data = persistent_data.write(key, value);\n opname = 'write';\n } else if (parsed_request.pathname.indexOf(\"/faa\") >= 0) {\n data = persistent_data.faa(key, value);\n opname = 'faa';\n } else if (parsed_request.pathname.indexOf(\"/cas\") >= 0) {\n opname = 'cas';\n var old_new_vals = value.split(',');\n if (old_new_vals[0] === '') { old_new_vals[0] = undefined; }\n if (old_new_vals[1] === '') { old_new_vals[1] = undefined; }\n data = persistent_data.cas(key, old_new_vals[0], old_new_vals[1]);\n } else {\n data = \"ERROR: No EMS command specified.\";\n }\n var datastr = JSON.stringify(data);\n console.log(\"Request #\" + requestN + \", slave process \" + myID + \": Op(\" +\n opname + \") key(\" + key + \") val(\" + value + \") data=\" + datastr);\n response.end(datastr);\n}\n\n\nif (cluster.isMaster) {\n /* The Master Process creates the EMS arrays that the slave processes\n * will attach to before creating the slave processes. This prevents\n * parallel hazards if all slaves tried creating the EMS arrays.\n */\n persistent_data_options.setFEtags = 'empty';\n persistent_data_options.doSetFEtags = true;\n persistent_data = ems.new(persistent_data_options);\n\n shared_counters_options.dataFill = 0;\n shared_counters_options.doDataFill = true;\n shared_counters_options.setFEtags = 'full';\n shared_counters_options.doSetFEtags = true;\n shared_counters = ems.new(shared_counters_options);\n\n // Seed the GUID generator with a unique starting value\n shared_counters.writeXF('GUID', Math.floor(Math.random() * 10000000));\n\n // All the one-time initialization is complete, now start slave processes.\n // The number of processes is the limit of the number of pending requests\n // before deadlock occurs.\n var nProcesses = Number(process.argv[2]) || 8; // Number of \"cluster\" processes to start\n for (var procnum = 0; procnum < nProcesses; procnum++) {\n cluster.fork();\n }\n\n console.log(\n \"REST API: ?key[=value]\\n\" +\n \" EMSCommand: read, readFE, readFF, writeXE, writeXF, writeEF, faa, cas\\n\" +\n \" key: The index into the EMS array, must be a valid JSON element\\n\" +\n \" value: For write commands, the value stored in EMS.\\n\" +\n \" For CAS, the value is the old and new values separated by a comma:\\n\" +\n \" /cas?key=oldval,newval\\n\");\n\n cluster.on('exit', function (worker, code, signal) {\n console.log(\"worker \" + worker.process.pid + \"died\");\n });\n} else {\n /* Each Slave Cluster processes must connect to the EMS array created\n * by the master process.\n */\n persistent_data_options.useExisting = true;\n persistent_data = ems.new(persistent_data_options);\n\n shared_counters_options.useExisting = true;\n shared_counters = ems.new(shared_counters_options);\n\n myID = shared_counters.faa('myID', 1);\n http.createServer(handleRequest).listen(port, function () {\n console.log(\"Server \" + myID + \" listening on: http://localhost:\" + port);\n });\n}\n"
},
{
"path": "Examples/README.md",
"content": "# Extended Memory Semantics (EMS) Examples\n#### Table of Contents\n* [Simple Loop Benchmarks](#Simple-Loop-Benchmarks)\n* [Key-Value Store](#kv_store.js)\n* [Work Queues](#Work-Queues)\n* [Parallel Web Server](#web_server.js)\n* [Word Counting](#Word-Count)\n* [Implied EMS Operations](#harmony_proxies.js)\n* [Inter-Language Programming](#Inter-language-Programming)\n\n## Simple Loop Benchmarks\n\nClick here for a __[Detailed Description of the EMS.js STREAMS implementation](https://github.com/SyntheticSemantics/ems/tree/master/Examples/STREAMS)__.\n\nThe original [STREAMS](https://www.cs.virginia.edu/stream/)\nbenchmark attempts to measure the _Speed Not To Exceed_ \nbandwidth of a CPU to it's attached RAM by performing large\nnumbers of simple operations on an array of integers or floats.\nThis version of STREAMS for EMS.js measures the rate of different atomic operations on\nshared integer data, it is implemented twice using two different parallelization schemes:\nBulk Synchronous Parallel (BSP), and Fork-Join (FJ).\n\nExperimental results are from the BSP version\n on an AWS `c4.8xlarge (132 ECUs, 36 vCPUs, 2.9 GHz, Intel Xeon E5-2666v3, 60 GiB memory`.\n\n
\n
\n\n\n## Parallelism Within a Single HTTP Request\nClick here for the [Web Server Parallelism Example](https://github.com/SyntheticSemantics/ems/tree/master/Examples/WebServer)\n\nEMS is a memory model and has no daemon process to monitor a network interface\nfor operations to execute.\nThis example builds on a standard HTTP server to implement\nshared memory parallelism within a single web request.\n\n## EMS as a Key Value Store\nClick here for the [EMS Key-Value Store Example](https://github.com/SyntheticSemantics/ems/tree/master/Examples/KeyValueStore)\n\nAn EMS array is used as a Key-Value store with a 1-to-1 mapping of REST operations \nto EMS memory operations.\nThe KV server is implemented using the Node.js built-in `Cluster` module to\ndemonstrate parallelism sources outside of EMS.\n\n\n## Work Queues\nThe parameters of the experiment are defined in the `initializeSharedData()` function.\nAn experiment creates several new EMS arrays and a work queue of \nrandomly generated transactions that update one or more of the arrays.\nParallel processes dequeue the work and perform atomic updates on \nthe new EMS arrays. Finally, the experiment performs checksums to ensure\nall the transactions were performed properly.\n\n```javascript\n arrLen = 1000000; // Maximum number of elements the EMS array can contain\n heapSize = 100000; // Amount of persistent memory to reserve for transactions\n nTransactions = 1000000; // Number of transactions to perform in the experiment\n nTables = 6; // Number of EMS arrays to perform transactions across\n maxNops = 5; // Maximum number of EMS arrays to update during a transaction\n```\n\n### `concurrent_Q_and_TM.js`\n```bash\nnode concurrent_Q_and_TM.js 4 # 1 process enqueues then processes work, 3 processes perform work\n````\nAll processes are started, one enqueues transactions while the others dequeue \nand perform the work. When all the work has been enqueued, that process also\nbegins dequeing and performing work.\n\n\n\n### `workQ_and_TM.js`\n```bash\nnode workQ_and_TM.js 4 # Enqueue all the work, all 4 processes deqeue and perform work\n````\nAll the transactions are enqueued, then all the processes begin \nto dequeue and perform work until no work remains in the queue.\n\n\n\n## [Word Counting](#Word-Count)\nMap-Reduce is often demonstrated using word counting because each document can\nbe processed in parallel, and the results of each document's dictionary reduced\ninto a single dictionary.\nIn contrast with Map-Reduce implementations using a \ndictionary per document, the EMS implementation \nmaintains a single shared dictionary, eliminating the need for a reduce phase.\n\nThe word counts are sorted in parallel and the most frequently appearing words\nare printed with their counts.\n\n\n### Word Counting Using Atomic Operations\nA parallel word counting program is implemented in Python and Javascript.\nA shared dictionary is used by all the tasks, updates are made using\natomic operations. Specifically, `Compare And Swap (CAS)` is used to\ninitialize a new word from undefined to a count of 1. If this CAS\nfails, the word already exists in the dictionary and the operation\nis retried using `Fetch and Add`, atomically incrementing the count.\n\n### Forming the \"Bag of Words\" with Word Counts\n```javascript\nvar file_list = fs.readdirSync(doc_path);\nvar splitPattern = new RegExp(/[ \\n,\\.\\\\/_\\-\\<\\>:\\;\\!\\@\\#\\$\\%\\&\\*\\(\\)=\\[\\]|\\\"\\'\\{\\}\\?\\—]/);\n// Iterate over all the files in parallel\nems.parForEach(0, file_list.length, function (fileNum) {\n var text = fs.readFileSync(doc_path + file_list[fileNum], 'utf8', \"r\");\n var words = text.replace(/[\\n\\r]/g, ' ').toLowerCase().split(splitPattern);\n // Sequentially iterate over all the words in one document\n words.forEach(function (word) {\n wordCounts.faa(word, 1); // Atomically increment the count of times this word was seen\n });\n});\n```\n\n### Sorting the Word Count list by Frequency\nParallel sorting of the word count is implemented as a multi-step process:\n\n1. The bag of words is processed by all the processess, each process\n building an ordered list of the most common words it encounters\n2. The partial lists from all the processes are concatenated into a single list\n3. The list of the most common words seen is sorted and truncated\n\n```javascript\nvar biggest_counts = new Array(local_sort_len).fill({\"key\": 0, \"count\": 0});\nems.parForEach(0, maxNKeys, function (keyN) {\n var key = wordCounts.index2key(keyN);\n if (key) {\n // Perform an insertion sort of the new key into the biggest_counts\n // list, deleting the last (smallest) element to preserve length.\n var keyCount = wordCounts.read(key);\n var idx = local_sort_len - 1;\n while (idx >= 0 && biggest_counts[idx].count < keyCount) { idx -= 1; }\n var newBiggest = {\"key\": key, \"count\": keyCount};\n if (idx < 0) {\n biggest_counts = [newBiggest].concat(biggest_counts.slice(0, biggest_counts.length - 1));\n } else if (idx < local_sort_len) {\n var left = biggest_counts.slice(0, idx + 1);\n var right = biggest_counts.slice(idx + 1);\n biggest_counts = left.concat([newBiggest].concat(right)).slice(0, -1);\n }\n }\n});\n// Concatenate all the partial (one per process) lists into one list\nstats.writeXF('most_frequent', []); // Initialize the list\nems.barrier(); // Wait for all procs to have finished initialization\nstats.writeEF('most_frequent', stats.readFE('most_frequent').concat(biggest_counts));\nems.barrier(); // Wait for all procs to have finished merge\nems.master(function() { // Sort & print the list of words, only one process is needed\n biggest_counts = stats.readFF('most_frequent');\n biggest_counts.sort(function (a, b) { return b.count - a.count; });\n // Print only the first \"local_sort_len\" items -- assume the worst case\n // of all the largest counts are discovered by a single process\n console.log(\"Most frequently appearing terms:\");\n for (var index = 0; index < local_sort_len; index += 1) {\n console.log(index + ': ' + biggest_counts[index].key + \" \" + biggest_counts[index].count);\n }\n});\n```\n\n### Word Count Performance\nThis section reports the results of running the Word Count example on\ndocuments from Project Gutenberg.\n2,981,712,952 words in several languages were parsed, totaling 12,664,852,220 bytes of text.\n\n\n\n\n\n## Inter-Language Programming\n[Inter-Language Shared Memory Programming](https://github.com/SyntheticSemantics/ems/tree/master/Examples/Interlanguage)\n\nThe programs `interlanguage.js` and `interlanguage.py` demonstrate sharing\nobjects between Javascript and Python.\nA variety of synchronization and execution models are shown.\n\n\n### Example\n\n\n###### Copyright (C)2011-2019 Jace A Mogill\n"
},
{
"path": "Examples/STREAMS/README.md",
"content": "# EMS.js STREAMS Bandwidth Benchmark\n\nThe original [STREAMS](https://www.cs.virginia.edu/stream/)\nbenchmark measures the _Speed Not To Exceed_ \nbandwidth of a CPU to it's attached RAM by performing large\nnumbers of simple operations. This version for\nEMS.js measures the rate of different atomic operations on\nshared integer data and is implemented using two different parallelization schemes:\nBulk Synchronous Parallel (BSP), and Fork-Join.\n\n\n## Parallel Execution Models\n\nThe EMS.js STREAMS benchmark is implemented twice, once using Bulk Synchronous Parallelism \nand again using Fork-Join parallelism. \n\nBSP goes parallel at program startup,\ndecomposes loop iterations across processes, and performs a barrier at the end\nof each loop. FJ starts serial and goes parallel for each loop, joining back\nto serial execution before the next loop.\n\n
\n
\n
\n
\n\n
\n\n
\n
\n
\n\n
\n
\n
\n\n\n## Running the Benchmark\n\nTo run either version of STREAMS benchmark, specify the number of processes\nas an argument to the program:\n\n```bash\nnode streams_bulk_sync_parallel.js 8\n```\n\nRunning the BSP version of the STREAMS benchmark reports the rate at\nwhich each loop is performed, each loop exercising a single\nEMS read and write primitive.\nA second set of \"Native\" loops performs similar operations on ordinary\nvolatile private memory, taking advantage of all possible native\noptimizations and just-in-time compilation.\nThese results show a performance difference of 20-50x \naccessing EMS memory versus native memory,\nwhich is still very favorable compared to the 1000-5000x difference accessing\na remote key-value store, or worse accessing magnetic media.\n\n\n```bash\ndunlin> time node streams_bulk_sync_parallel.js 3\nEMStask 0: 1,000,000 write 6,896,551 ops/sec\nEMStask 0: 1,000,000 writeXE 7,246,376 ops/sec\nEMStask 0: 1,000,000 writeXF 7,246,376 ops/sec\nEMStask 0: 1,000,000 read 7,042,253 ops/sec\nEMStask 0: 1,000,000 reread 6,896,551 ops/sec\nEMStask 0: 1,000,000 readFF 7,299,270 ops/sec\nEMStask 0: 1,000,000 readFE 7,194,244 ops/sec\nEMStask 0: 1,000,000 writeEF 7,299,270 ops/sec\nEMStask 0: 1,000,000 copy 3,521,126 ops/sec\nEMStask 0: 1,000,000 recopy 3,484,320 ops/sec\nEMStask 0: 1,000,000 c=a*b 2,364,066 ops/sec\nEMStask 0: 1,000,000 c+=a*b 1,715,265 ops/sec\nEMStask 0: 1,000,000 checksum Infinity ops/sec\n------------------------ START NATIVE ARRAY\nEMStask 0: 1,000,000 write 333,333,333 ops/sec\nEMStask 0: 1,000,000 rd/write 250,000,000 ops/sec\nEMStask 0: 1,000,000 read 125,000,000 ops/sec\nEMStask 0: 1,000,000 reread 250,000,000 ops/sec\nEMStask 0: 1,000,000 copy 200,000,000 ops/sec\nEMStask 0: 1,000,000 rmwcopy 250,000,000 ops/sec\nEMStask 0: 1,000,000 c=a*b 100,000,000 ops/sec\nEMStask 0: 1,000,000 c+=a*b 142,857,142 ops/sec\nEMStask 0: 1,000,000 sum Infinity ops/sec\n\nreal\t0m2.938s\nuser\t0m8.190s\nsys\t0m0.172s\n```\n\n__NOTE:__ Adding timing instrumentation to the Fork-Join version is\nleft as an exercise to the reader. \n\n\n## Benchmark Results\nBenchmark results for the BSP implementation on an AWS\n`c4.8xlarge (132 ECUs, 36 vCPUs, 2.9 GHz, Intel Xeon E5-2666v3, 60 GiB memory`.\n\n
\n
\n\n"
},
{
"path": "Examples/STREAMS/streams_bulk_sync_parallel.js",
"content": "/*-----------------------------------------------------------------------------+\n | Extended Memory Semantics (EMS) Version 1.0.0 |\n | Synthetic Semantics http://www.synsem.com/ mogill@synsem.com |\n +-----------------------------------------------------------------------------+\n | Copyright (c) 2011-2014, Synthetic Semantics LLC. All rights reserved. |\n | Copyright (c) 2015-2016, Jace A Mogill. All rights reserved. |\n | |\n | Redistribution and use in source and binary forms, with or without |\n | modification, are permitted provided that the following conditions are met: |\n | * Redistributions of source code must retain the above copyright |\n | notice, this list of conditions and the following disclaimer. |\n | * Redistributions in binary form must reproduce the above copyright |\n | notice, this list of conditions and the following disclaimer in the |\n | documentation and/or other materials provided with the distribution. |\n | * Neither the name of the Synthetic Semantics nor the names of its |\n | contributors may be used to endorse or promote products derived |\n | from this software without specific prior written permission. |\n | |\n | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS |\n | \"AS IS\" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT |\n | LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR |\n | A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL SYNTHETIC |\n | SEMANTICS LLC BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, |\n | EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, |\n | PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR |\n | PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF |\n | LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING |\n | NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS |\n | SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. |\n | |\n +-----------------------------------------------------------------------------*/\n//\n// Based on John D. McCalpin's (Dr. Bandwdith) STREAMS benchmark:\n// https://www.cs.virginia.edu/stream/\n//\n//\n// Also see the Fork-Join version in the Examples directory\n// ===============================================================================\nvar ems = require('ems')(parseInt(process.argv[2]), false);\nvar arrLen = 1000000;\nvar a = ems.new(arrLen);\nvar b = ems.new(arrLen);\nvar c = ems.new(arrLen);\nvar x = new Array(arrLen);\nvar y = new Array(arrLen);\nvar z = new Array(arrLen);\n\n\n//-------------------------------------------------------------------\n// Timer function\nfunction stopTimer(timer, nOps, label) {\n function fmtNumber(n) {\n var s = ' ' + n.toString().replace(/\\B(?=(\\d{3})+(?!\\d))/g, \",\");\n if(n < 1) return n;\n else { return s.substr(s.length - 15, s.length); }\n }\n ems.master( function() {\n var now = new Date().getTime();\n var x = (nOps*1000000) / ((now - timer) *1000);\n ems.diag(fmtNumber(nOps) + label + fmtNumber(Math.floor(x).toString()) + \" ops/sec\");\n } );\n}\n\n\nvar timeStart = new Date().getTime();\nems.parForEach(0, arrLen, function(idx) { a.write(idx, idx) } );\nstopTimer(timeStart, arrLen, \" write \");\n\ntimeStart = new Date().getTime();\nems.parForEach(0, arrLen, function(idx) { a.writeXE(idx, idx) } );\nstopTimer(timeStart, arrLen, \" writeXE \");\n\ntimeStart = new Date().getTime();\nems.parForEach(0, arrLen, function(idx) { a.writeXF(idx, idx) } );\nstopTimer(timeStart, arrLen, \" writeXF \");\n\ntimeStart = new Date().getTime();\nems.parForEach(0, arrLen, function(idx) { a.read(idx, idx) } );\nstopTimer(timeStart, arrLen, \" read \");\n\ntimeStart = new Date().getTime();\nems.parForEach(0, arrLen, function(idx) { a.read(idx, idx) } );\nstopTimer(timeStart, arrLen, \" reread \");\n\ntimeStart = new Date().getTime();\nems.parForEach(0, arrLen, function(idx) { a.readFF(idx, idx) } );\nstopTimer(timeStart, arrLen, \" readFF \");\n\ntimeStart = new Date().getTime();\nems.parForEach(0, arrLen, function(idx) { a.readFE(idx, idx) } );\nstopTimer(timeStart, arrLen, \" readFE \");\n\ntimeStart = new Date().getTime();\nems.parForEach(0, arrLen, function(idx) { a.writeEF(idx, idx) } );\nstopTimer(timeStart, arrLen, \" writeEF \");\n\ntimeStart = new Date().getTime();\nems.parForEach(0, arrLen, function(idx) { b.writeXF(idx, a.readFF(idx)) } );\nstopTimer(timeStart, arrLen, \" copy \");\n\ntimeStart = new Date().getTime();\nems.parForEach(0, arrLen, function(idx) { b.writeXF(idx, a.readFF(idx)) } );\nstopTimer(timeStart, arrLen, \" recopy \");\n\ntimeStart = new Date().getTime();\nems.parForEach(0, arrLen, function(idx) { c.writeXF(idx, a.readFF(idx) * b.readFF(idx)) } );\nstopTimer(timeStart, arrLen, \" c=a*b \");\n\ntimeStart = new Date().getTime();\nems.parForEach(0, arrLen, function(idx) { c.writeXF(idx, c.readFF(idx) + (a.readFF(idx) * b.readFF(idx))) } );\nstopTimer(timeStart, arrLen, \" c+=a*b \");\n\ntimeStart = new Date().getTime();\nstopTimer(timeStart, arrLen, \" checksum \");\n\n\n//===========================================================================================\nvar idx;\n\nif(ems.myID == 0) {\n console.log('------------------------ START NATIVE ARRAY');\n timeStart = new Date().getTime();\n for(idx = 0; idx < arrLen; idx++ ) { x[idx] = idx }\n stopTimer(timeStart, arrLen, \" write \");\n\n timeStart = new Date().getTime();\n for(idx = 0; idx < arrLen; idx++ ) { x[idx] += idx }\n stopTimer(timeStart, arrLen, \" rd/write\");\n\n timeStart = new Date().getTime();\n var dummy = 0;\n for(idx = 0; idx < arrLen; idx++ ) { dummy += x[idx] }\n stopTimer(timeStart, arrLen, \" read \");\n\n timeStart = new Date().getTime();\n for(idx = 0; idx < arrLen; idx++ ) { dummy += x[idx] }\n stopTimer(timeStart, arrLen, \" reread \");\n\n timeStart = new Date().getTime();\n for(idx = 0; idx < arrLen; idx++ ) { y[idx] = x[idx] }\n stopTimer(timeStart, arrLen, \" copy \");\n\n timeStart = new Date().getTime();\n for(idx = 0; idx < arrLen; idx++ ) { y[idx] += x[idx] }\n stopTimer(timeStart, arrLen, \" rmwcopy \");\n\n timeStart = new Date().getTime();\n for(idx = 0; idx < arrLen; idx++ ) { z[idx] = x[idx] * y[idx] }\n stopTimer(timeStart, arrLen, \" c=a*b \");\n\n timeStart = new Date().getTime();\n for(idx = 0; idx < arrLen; idx++ ) { z[idx] += x[idx] * y[idx] }\n stopTimer(timeStart, arrLen, \" c+=a*b \");\n\n timeStart = new Date().getTime();\n stopTimer(timeStart, arrLen, \" sum \");\n}\n\n"
},
{
"path": "Examples/STREAMS/streams_fork_join.js",
"content": "/*-----------------------------------------------------------------------------+\n | Extended Memory Semantics (EMS) Version 1.0.0 |\n | Synthetic Semantics http://www.synsem.com/ mogill@synsem.com |\n +-----------------------------------------------------------------------------+\n | Copyright (c) 2011-2014, Synthetic Semantics LLC. All rights reserved. |\n | Copyright (c) 2015-2016, Jace A Mogill. All rights reserved. |\n | |\n | Redistribution and use in source and binary forms, with or without |\n | modification, are permitted provided that the following conditions are met: |\n | * Redistributions of source code must retain the above copyright |\n | notice, this list of conditions and the following disclaimer. |\n | * Redistributions in binary form must reproduce the above copyright |\n | notice, this list of conditions and the following disclaimer in the |\n | documentation and/or other materials provided with the distribution. |\n | * Neither the name of the Synthetic Semantics nor the names of its |\n | contributors may be used to endorse or promote products derived |\n | from this software without specific prior written permission. |\n | |\n | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS |\n | \"AS IS\" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT |\n | LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR |\n | A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL SYNTHETIC |\n | SEMANTICS LLC BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, |\n | EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, |\n | PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR |\n | PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF |\n | LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING |\n | NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS |\n | SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. |\n | |\n +-----------------------------------------------------------------------------*/\n//\n// Based on John D. McCalpin's (Dr. Bandwdith) STREAMS benchmark:\n// https://www.cs.virginia.edu/stream/\n//\n// Also see the Bulk Synchronous Parallel implementation in the Examples directory\n// ===============================================================================\n'use strict';\nvar ems = require('ems')(parseInt(process.argv[2]), true, 'fj');\n\nems.parallel( function() {\n var arrLen = 1000000;\n var array_a = ems.new(arrLen);\n var array_b = ems.new(arrLen);\n var array_c = ems.new(arrLen);\n\n ems.parForEach(0, arrLen, function(idx) { array_a.write(idx, idx)});\n ems.parForEach(0, arrLen, function(idx) { array_a.writeXE(idx, idx)});\n ems.parForEach(0, arrLen, function(idx) { array_a.writeXF(idx, idx)});\n ems.parForEach(0, arrLen, function(idx) { array_a.read(idx, idx)});\n ems.parForEach(0, arrLen, function(idx) { array_a.readFF(idx, idx)});\n ems.parForEach(0, arrLen, function(idx) { array_a.readFE(idx, idx)});\n ems.parForEach(0, arrLen, function(idx) { array_a.writeEF(idx, idx)});\n ems.parForEach(0, arrLen, function(idx) { array_b.writeXF(idx, array_a.readFF(idx))});\n ems.parForEach(0, arrLen, function(idx) { array_c.writeXF(idx, array_a.readFF(idx) * array_b.readFF(idx))});\n ems.parForEach(0, arrLen, function(idx) { array_c.writeXF(idx, array_c.readFF(idx) + (array_a.readFF(idx) * array_b.readFF(idx)))});\n} );\n\n// Fork-Join tasks must exit themselves, otherwise they will wait indefinitely for more work\nprocess.exit(0);\n"
},
{
"path": "Examples/WebServer/README.md",
"content": "# Parallelism Generating a Single HTTP Response\n\nUsing Fork-Join (FJ) parallelism a conventional \nHTTP server can use multiple cores to generate\na single a response by forking threads\nwhen a request is received, generating the response in\nparallel, joining all the threads when the work is complete,\nand then returning the response from the original single thread.\n\nHTTP applications frequently require state to persist between requests,\nthis example uses the URL path as a session ID that can be used to\npersist state between requests. Each request itself may include\none or more fork-join parallel regions.\n\n## Parallel Web Server Architecture\n\nThis example combines Node.js' built-in HTTP server module\nwith EMS for shared memory and task level parallelism.\n\nWhen the server is started it allocates a globally shared\nEMS shared memory (`serverState.ems`) that exists until the server exits.\nThe global shared memory is used for diagnostic counters like the \nnumber of queries as well as key-value pairs that can be read\nor written from a parallel or serial region of any request or session.\n\n\n\n## REST API\n\nA valid request URL consists of path naming the session ID\nand a query that specifies if the session is new/restarting, or\nis a continuation of a previous session.\n \n__REST API:__ `/sessionID?&`\n \n- `sessionID` - An unique session identifier. The shared memory\n persistent file will be named `session.sessionID.ems` and stored\n in the current working directory.\n\n- `?` - A `new` session will remove an existing\n session-private shared memory if present, an `old` session\n will attach to a previously created shared memory that was\n persisted to disk.\n \n- `?` - Release or preserve, respectively, the \n session's shared memory at the end of the request.\n\n\n## Fork-Join Parallelism\n\nA process using Fork-Join parallelism starts single-threaded,\ncreating parallel regions when needed, then returning to\nsingle-threaded execution when all the threads in the \nparallel region have completed execution.\nEMS amortizes the cost of starting new processes by using a thread pool\nwhere idle threads can quickly be restarted when they are needed again.\n\nBy way of comparison, all the processes in Bulk Synchronous Parallelism\nare always executing, barriers can be used to synchronize execution.\n\n
\n
\n
\n
\n Fork-Join Parallelism \n \n
\n
\n
\n
\n Bulk Synchronous Parallelism \n \n
\n
\n
\n
\n\n\n## Web Server Process\n\nThe web server is a conventional single-threaded based on Node.js'\nbuilt-in HTTP module. As part of the single-threaded initialization\nEMS is initialized for Fork-Join parallelism and a shared memory\nis created for the processes to share diagnostic counters and other\ncommon state. This shared memory lasts for the life of the web\nserver process and does not persist between invocations of the server.\n\nWhen a request is received, the URL is parsed, then the server enters\na parallel region. From the parallel region each process creates or\nattaches to shared memory created \n\n\n\n\n\n## Example\nStart the server:\n`node webServer.js [# of processes, default=8]`\n\nFrom another terminal or browser, issue requests to the server.\n\nA parallel region is created for each request received by the web server,\nhandling the single request with multiple processes.\nDifferent variables in the response are used to illustrate\ndifferent types of data updates.\n\n- `requestN` - Every request is given an unique ID\n- `sessionData` - Each process records it's \"work\" in the private EMS\n memory allocated for the session (`foo` or `bar`). \n The results in the form\n `(Req:/)`, the processes may complete\n in any order, but each request is handled sequentially.\n- `bignum` - The sum of a loop incrementing a counter in session\n private EMS memory\n- `randomSum` - The sum of random numbers generated by all the processes,\n stored in the shared global EMS memory \n \nFollowing the join at the end of the parallel region,\n single-threaded execution continues, copying the different\n variables into a response object \n\n\n```\ndunlin> curl 'http://localhost:8080/foo?new&keep' # Create a new persistent memory for session \"foo\"\n{\"requestN\":0,\"sessionData\":\"0(Req:0/2) (Req:0/7) (Req:0/1) (Req:0/4) (Req:0/5) (Req:0/3) (Req:0/0) (Req:0/6) \",\"bignum\":100000,\"randomSum\":6103}\n\ndunlin> curl 'http://localhost:8080/foo?old&keep' # Modify the existing \"foo\" memory\n{\"requestN\":1,\"sessionData\":\"0(Req:0/2) (Req:0/7) (Req:0/1) (Req:0/4) (Req:0/5) (Req:0/3) (Req:0/0) (Req:0/6) (Req:1/0) (Req:1/4) (Req:1/1) (Req:1/3) (Req:1/5) (Req:1/7) (Req:1/6) (Req:1/2) \",\"bignum\":200000,\"randomSum\":16670}\n\ndunlin> curl 'http://localhost:8080/bar?old' # Error because \"bar\" does not yet exist, 'keep' implied\nERROR: Requested \"old\" shared memory bar does not already exist.\n\ndunlin> curl 'http://localhost:8080/bar?new&free' # Create and free session memory \"bar\"\n{\"requestN\":3,\"sessionData\":\"0(Req:3/4) (Req:3/7) (Req:3/5) (Req:3/3) (Req:3/0) (Req:3/6) (Req:3/2) (Req:3/1) \",\"bignum\":100000,\"randomSum\":25530}\n\ndunlin> curl 'http://localhost:8080/foo?old&keep' # Modify the existing \"foo\" memory again\n{\"requestN\":4,\"sessionData\":\"0(Req:0/2) (Req:0/7) (Req:0/1) (Req:0/4) (Req:0/5) (Req:0/3) (Req:0/0) (Req:0/6) (Req:1/0) (Req:1/4) (Req:1/1) (Req:1/3) (Req:1/5) (Req:1/7) (Req:1/6) (Req:1/2) (Req:4/0) (Req:4/5) (Req:4/2) (Req:4/7) (Req:4/6) (Req:4/1) (Req:4/3) (Req:4/4) \",\"bignum\":300000,\"randomSum\":30114}\n\ndunlin> curl 'http://localhost:8080/foo?new&free' # Modify the existing \"foo\" memory again, this time freeing it\n{\"requestN\":5,\"sessionData\":\"0(Req:5/4) (Req:5/7) (Req:5/5) (Req:5/0) (Req:5/2) (Req:5/6) (Req:5/1) (Req:5/3) \",\"bignum\":100000,\"randomSum\":36887}\n\ndunlin> curl 'http://localhost:8080/foo?old&free' # The memory \"foo\" has been released\nERROR: Requested \"old\" shared memory foo does not already exist.\n```\n\n\n###### Copyright (C)2017 Jace A Mogill - All Rights Reserved.\n"
},
{
"path": "Examples/WebServer/webServer.js",
"content": "/*-----------------------------------------------------------------------------+\n | Extended Memory Semantics (EMS) Version 1.4.5 |\n | Synthetic Semantics http://www.synsem.com/ mogill@synsem.com |\n +-----------------------------------------------------------------------------+\n | Copyright (c) 2011-2014, Synthetic Semantics LLC. All rights reserved. |\n | Copyright (c) 2015-2017, Jace A Mogill. All rights reserved. |\n | |\n | Redistribution and use in source and binary forms, with or without |\n | modification, are permitted provided that the following conditions are met: |\n | * Redistributions of source code must retain the above copyright |\n | notice, this list of conditions and the following disclaimer. |\n | * Redistributions in binary form must reproduce the above copyright |\n | notice, this list of conditions and the following disclaimer in the |\n | documentation and/or other materials provided with the distribution. |\n | * Neither the name of the Synthetic Semantics nor the names of its |\n | contributors may be used to endorse or promote products derived |\n | from this software without specific prior written permission. |\n | |\n | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS |\n | \"AS IS\" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT |\n | LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR |\n | A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL SYNTHETIC |\n | SEMANTICS LLC BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, |\n | EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, |\n | PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR |\n | PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF |\n | LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING |\n | NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS |\n | SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. |\n | |\n +-----------------------------------------------------------------------------*/\n'use strict';\n// Number of processes to use in parallel regions\nvar nProcs = process.argv[2] || 8;\n// Initialize EMS for FJ parallelism\nvar ems = require('ems')(parseInt(nProcs), true, 'fj');\n// Load modules for dealing with HTTP\nvar http = require('http');\n// Load modules for parsing a request URL\nvar url = require('url');\n// Port number to which the server listens\nvar port = 8080;\n// Global, persistent, shared memory that is visible from any process during any HTTP request callback\nvar globallySharedData;\n\n\n/**\n * Fork into a parallel region to initialize each parallel processes\n * by attaching the shared server status and loading 'require'd modules\n */\nems.parallel(function () {\n globallySharedData = ems.new({\n dimensions: [1000],\n heapSize: [10000],\n useExisting: false,\n useMap: true,\n doDataFill: true,\n dataFill: 0,\n filename: 'serverState.ems'\n });\n});\n\n\n\n// Shared memory that last for a session, global so it is scoped to last more more than one request\nvar sessionSharedData;\n\n\n/**\n * When an HTTP request arrives this callback is executed.\n * From within that callback, parallel processes are forked\n * to work on the response, then join together back to the original\n * callback thread from where the response is issued.\n * @param request - Request object with URL\n * @param response - Object used respond to the request\n */\nfunction handleRequest(request, response) {\n // Increment (Fetch And Add) the number of requests counter\n var requestN = globallySharedData.faa('nRequests', 1);\n // Request URL parsed into UTF8\n var parsedRequest = url.parse(request.url, true);\n // Session ID part of the request path\n var key = parsedRequest.pathname.split(/\\//g)[1];\n\n if(!key) {\n response.end('ERROR: Incorrectly formatted or missing session unique ID (' + parsedRequest.pathname + ')');\n } else {\n // Fork a parallel region\n ems.parallel(parsedRequest, requestN, key, (parsedRequest, requestN, key) => {\n /// EMS options for the session's shared memory\n var sessionData = {\n dimensions: [100], // Up to 100 keys in this memory\n heapSize: [100000], // At least 100KB of memory\n useMap: true, // Index using any type of key, not linear integer indexing\n filename: 'session.' + key + '.ems', // Filename of persistent memory image\n };\n\n // Set EMS memory options to create a new or attach to an existing persistent EMS memory\n if (parsedRequest.query.new === \"\") {\n sessionData.dataFill = 0;\n sessionData.doDataFill = true;\n sessionData.useExisting = false;\n sessionData.doSetFEtags = true;\n sessionData.setFEtags = 'full';\n } else if (parsedRequest.query.old === \"\") {\n sessionData.useExisting = true;\n } else {\n // Send only one response, but all tasks exit the parallel region\n if (ems.myID === 0) {\n response.end('ERROR: Invalid new/old state (' + JSON.stringify(parsedRequest.query) + ') for private shared memory');\n }\n return;\n }\n\n // Create or attach to existing EMS memory\n sessionSharedData = ems.new(sessionData);\n if (!sessionSharedData) {\n // Send only one response, but all tasks exit the parallel region\n if (ems.myID === 0) {\n response.end('ERROR: Requested \"old\" shared memory ' + key + ' does not already exist.');\n }\n return;\n }\n\n // Each task generates it's own random number and performs all the iterations.\n var sum = 0;\n for(var idx = 0; idx < Math.floor(Math.random() * 1000); idx += 1) {\n sum += idx;\n }\n // All processes accumulate (Fetch And Add) directly into globally shared counter\n globallySharedData.faa('sum', sum);\n\n // Distribute (ie: map) the iterations of a loop across all the processes\n ems.parForEach(0, 100000, function(idx) {\n sessionSharedData.faa('bignum', 1); // Accumulate into a per-session counter\n });\n\n // Each process appends it's results to the session's EMS memory using Fetch And Add.\n // Note that the memory was initialized with 0 to which this text is appended\n sessionSharedData.faa('results', \"(Req:\" + requestN + \"/\" + ems.myID + \") \");\n });\n // All parallel processes have returned before the serial region resumes here\n\n /// Construct a return object containing data from both the globally shared memory\n /// and session-private data.\n var resp = { 'requestN' : requestN };\n // Parallel region may have exited due to error, in which case there are no results to return\n if (sessionSharedData) {\n resp.sessionData = sessionSharedData.readFF('results');\n resp.bignum = sessionSharedData.readFF('bignum');\n resp.randomSum = globallySharedData.readFF('sum');\n response.end(JSON.stringify(resp));\n\n if (parsedRequest.query.free === \"\") {\n sessionSharedData.destroy(true);\n }\n } else {\n response.end('ERROR: Parallel region exited without creating/accessing session shared memory');\n }\n }\n}\n\n\n// Create the Web server\nhttp.createServer(handleRequest).listen(port, function () {\n console.log(\n \"REST API: /UID?[old|new]&[keep|free]\\n\" +\n \" /UID Unique session identifier\\n\" +\n \" ?[old|new] A 'new' session will remove and create a new named private shared memory\\n\" +\n \" ?[keep|free] Release or preserve, respectively, the named private shared memory\\n\" +\n \"\\nExamples:\\n\" +\n \" curl http://localhost:8080/foo?new&keep # Create a new persistent memory for session \\\"foo\\\"\\n\" +\n \" curl http://localhost:8080/foo?old&keep # Modify the existing \\\"foo\\\" memory\\n\" +\n \" curl http://localhost:8080/bar?old # Error because \\\"bar\\\" does not yet exist\\n\" +\n \" curl http://localhost:8080/bar?new&free # Create and free session memory \\\"bar\\\"\\n\" +\n \"\\nServer listening on: http://localhost:\" + port + \"\\n\"\n );\n});\n"
},
{
"path": "Examples/concurrent_Q_and_TM.js",
"content": "/*-----------------------------------------------------------------------------+\n | Extended Memory Semantics (EMS) Version 1.0.0 |\n | Synthetic Semantics http://www.synsem.com/ mogill@synsem.com |\n +-----------------------------------------------------------------------------+\n | Copyright (c) 2011-2014, Synthetic Semantics LLC. All rights reserved. |\n | Copyright (c) 2015-2016, Jace A Mogill. All rights reserved. |\n | |\n | Redistribution and use in source and binary forms, with or without |\n | modification, are permitted provided that the following conditions are met: |\n | * Redistributions of source code must retain the above copyright |\n | notice, this list of conditions and the following disclaimer. |\n | * Redistributions in binary form must reproduce the above copyright |\n | notice, this list of conditions and the following disclaimer in the |\n | documentation and/or other materials provided with the distribution. |\n | * Neither the name of the Synthetic Semantics nor the names of its |\n | contributors may be used to endorse or promote products derived |\n | from this software without specific prior written permission. |\n | |\n | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS |\n | \"AS IS\" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT |\n | LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR |\n | A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL SYNTHETIC |\n | SEMANTICS LLC BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, |\n | EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, |\n | PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR |\n | PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF |\n | LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING |\n | NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS |\n | SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. |\n | |\n +-----------------------------------------------------------------------------+\n | Program Description: |\n | |\n | Sequentially enqueue transactions from thread 0 while all other |\n | threads consume them. When all the operations have been queued |\n | thread 0 also begins dequeing work and processing transactions. |\n | |\n | Unlike the workQ_and_TM.js example, randomInRange() is only called |\n | from thread 0 so it is defined only in the main thread's context. |\n | |\n +-----------------------------------------------------------------------------*/\n'use strict';\n// Initialize EMS to use the fork-join execution model\nvar ems = require('ems')(parseInt(process.argv[2]), true, 'fj');\nvar assert = require('assert');\nvar workQ;\nvar totalNops;\nvar checkNops;\nvar arrLen;\nvar heapSize;\nvar nTransactions;\nvar nTables;\nvar maxNops;\nvar tables;\n\n//---------------------------------------------------------------------------\n// Generate a random integer within a range (inclusive) from 'low' to 'high'\n//\nfunction randomInRange(low, high) {\n return (Math.floor((Math.random() * (high - low)) + low));\n}\n\n//-------------------------------------------------------------------\n// Timer functions\nfunction timerStart() {\n return new Date().getTime();\n}\n\nfunction timerStop(timer, nOps, label, myID) {\n function fmtNumber(n) {\n var s = ' ' + n.toString().replace(/\\B(?=(\\d{3})+(?!\\d))/g, \",\");\n if (n < 1) return n;\n else {\n return s.substr(s.length - 15, s.length);\n }\n }\n\n var now = new Date().getTime();\n var opsPerSec = (nOps * 1000000) / ((now - timer) * 1000);\n if (typeof myID === undefined || myID === 0) {\n console.log(fmtNumber(nOps) + label + fmtNumber(Math.floor(opsPerSec).toString()) + \" ops/sec\");\n }\n}\n\n\n//---------------------------------------------------------------------------\n// Initialize shared data: global scalars, EMS buffers for statistics\n// and checksums, an EMS array to be used as a work queue, and many\n// tables to perform transactions on.\n//\nfunction initializeSharedData() {\n arrLen = 1000000; // Maximum number of elements the EMS array can contain\n heapSize = 100000; // Amount of persistent memory to reserve for transactions\n nTransactions = 1000000; // Number of transactions to perform in the experiment\n nTables = 6; // Number of EMS arrays to perform transactions across\n maxNops = 5; // Maximum number of EMS arrays to update during a transaction\n tables = [];\n totalNops = ems.new(2);\n checkNops = ems.new(1);\n\n //---------------------------------------------------------------------------\n // The queue of transactions to perform\n // [ table#, index, read-only ]\n workQ = ems.new({\n dimensions: [nTransactions + ems.nThreads],\n heapSize: nTransactions * 200,\n useExisting: false,\n setFEtags: 'empty'\n });\n\n //---------------------------------------------------------------------------\n // Create all the tables\n for (var tableN = 0; tableN < nTables; tableN++) {\n tables[tableN] = ems.new({\n dimensions: [arrLen],\n heapSize: 0,\n useExisting: false,\n filename: '/tmp/EMS_tm' + tableN,\n doDataFill: true,\n dataFill: 0,\n setFEtags: 'full'\n });\n }\n}\n\n\n//---------------------------------------------------------------------------\n// Create 'nTransactions' many transactions, each having a random number (up\n// to maxNops) randomly chosen values, some of which are read-only.\n// Because tables and elements are chosen at random, it is necessary to\n// remove duplicate elements in a single transaction, to prevent deadlocks.\n//\n// Transactions are pushed onto a shared queue\nfunction generateTransactions() {\n //---------------------------------------------------------------------------\n // Generate operations involving random elements in random EMS arrays\n // and enqueue them on the work queue\n for (var transN = 0; transN < nTransactions; transN++) {\n var ops = [];\n var nOps = randomInRange(1, maxNops);\n // var indexes = [];\n for (var opN = 0; opN < nOps; opN++) {\n var tableN = randomInRange(0, nTables);\n var idx = randomInRange(0, arrLen);\n if (transN % 2 == 0 || opN % 3 > 0) {\n ops.push([tableN, idx, true]);\n } else {\n ops.push([tableN, idx]);\n }\n }\n\n // De-duplicate operands in a single transaction as that would deadlock\n var indicies = [];\n var uids = [];\n for (var i = 0; i < ops.length; ++i) {\n indicies[i] = i;\n uids[i] = (ops[i][0] * 1000000000000) + ops[i][1];\n }\n var uniq = [];\n for (opN = 0; opN < ops.length; opN++) {\n var isDupl = false;\n for (var checkN = 0; checkN < ops.length; checkN++) {\n if (opN != checkN && uids[opN] == uids[checkN]) {\n isDupl = true;\n break;\n }\n }\n if (!isDupl) {\n uniq.push(ops[opN]);\n }\n }\n workQ.enqueue(JSON.stringify(uniq));\n }\n\n}\n\n\n//------------------------------------------------------------------\n// Simultaneously generate and consume transactions\n// Thread 0 enqueues them while the rest dequeue transactions.\n// When thread 0 finishes creating new work, it also begins to\n// dequeue transactions from the workQ.\n//\n// If there is nothing on the queue, keep checking until the \"DONE\"\n// message is dequeued.\n//\nfunction performTransactions() {\n //------------------------------------------------------------------\n // Generate the transactions concurrently with their consumption\n if (ems.myID == 0) {\n var startTime = timerStart();\n generateTransactions();\n\n // After all the work has been enqueued, add DONE semaphores to the\n // end of the queue so they are processed only after all the work\n // has been issued. Each thread enqueues one event and can only\n // consume one before exiting.\n for (var taskN = 0; taskN < ems.nThreads; taskN++) {\n workQ.enqueue(\"DONE\");\n }\n timerStop(startTime, nTransactions, \" transactions enqueued \", ems.myID);\n }\n\n\n //------------------------------------------------------------------\n // Process the transactions\n var rwNops = 0;\n var readNops = 0;\n\n while (true) {\n var str = workQ.dequeue();\n if (str !== undefined) {\n if (str === \"DONE\") {\n break\n } else {\n var ops = JSON.parse(str);\n for (var opN = 0; opN < ops.length; opN++) {\n ops[opN][0] = tables[ops[opN][0]];\n }\n var transaction = ems.tmStart(ops);\n ops.forEach(function (op) {\n var tmp = op[0].read(op[1]);\n if (op[2] != true) {\n rwNops++;\n op[0].write(op[1], tmp + 1);\n } else {\n readNops++;\n }\n });\n ems.tmEnd(transaction, true);\n }\n } else {\n // Queue was empty, but have not yet seen DONE event, so\n // keep trying to dequeue work\n }\n }\n totalNops.faa(0, rwNops);\n totalNops.faa(1, readNops);\n}\n\n\n\n//=======================================================================\n// B E G I N P R O G R A M M A I N\n//-------------------------------------------------------\n// Initialize the shared data\nvar startTime = timerStart();\nems.parallel(initializeSharedData);\ntimerStop(startTime, nTables, \" tables initialized \", ems.myID);\n\n// Data to be initialized only once\ntotalNops.writeXF(0, 0);\ntotalNops.writeXF(1, 0);\ncheckNops.writeXF(0, 0);\n\n\n// Perform all the transactions\nstartTime = timerStart();\nems.parallel(performTransactions);\ntimerStop(startTime, nTransactions, \" transactions performed\", ems.myID);\ntimerStop(startTime, totalNops.readFF(0), \" table updates \", ems.myID);\ntimerStop(startTime, totalNops.readFF(0) + totalNops.readFF(1), \" elements referenced \", ems.myID);\n\n\n//------------------------------------------------------------------\n// Sum all the values in the tables to account for all the transactions\nstartTime = timerStart();\nems.parallel(function () {\n ems.parForEach(0, nTables, function (tableN) {\n var localSum = 0;\n for (var idx = 0; idx < arrLen; idx++) {\n localSum += tables[tableN].read(idx);\n }\n checkNops.faa(0, localSum);\n });\n});\ntimerStop(startTime, nTables * arrLen, \" elements checked \", ems.myID);\n\nassert(checkNops.readFF(0) == totalNops.readFF(0),\n \"Error in final sum = \" + checkNops.readFF(0) + \" should be=\" + totalNops.readFF(0));\nconsole.log(\"Results are correct\");\n// One liner to get all processes to exit\nems.parallel(function () { process.exit(0) });\n"
},
{
"path": "Examples/wordCount.js",
"content": "/*-----------------------------------------------------------------------------+\n | Extended Memory Semantics (EMS) Version 1.4.0 |\n | Synthetic Semantics http://www.synsem.com/ mogill@synsem.com |\n +-----------------------------------------------------------------------------+\n | Copyright (c) 2011-2014, Synthetic Semantics LLC. All rights reserved. |\n | Copyright (c) 2015-2016, Jace A Mogill. All rights reserved. |\n | |\n | Redistribution and use in source and binary forms, with or without |\n | modification, are permitted provided that the following conditions are met: |\n | * Redistributions of source code must retain the above copyright |\n | notice, this list of conditions and the following disclaimer. |\n | * Redistributions in binary form must reproduce the above copyright |\n | notice, this list of conditions and the following disclaimer in the |\n | documentation and/or other materials provided with the distribution. |\n | * Neither the name of the Synthetic Semantics nor the names of its |\n | contributors may be used to endorse or promote products derived |\n | from this software without specific prior written permission. |\n | |\n | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS |\n | \"AS IS\" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT |\n | LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR |\n | A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL SYNTHETIC |\n | SEMANTICS LLC BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, |\n | EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, |\n | PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR |\n | PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF |\n | LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING |\n | NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS |\n | SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. |\n | |\n +-----------------------------------------------------------------------------*/\n'use strict';\nvar ems = require('ems')(parseInt(process.argv[2]));\nvar fs = require('fs');\n\n\n//-------------------------------------------------------------------\n// Allocate the word count dictionary\nvar maxNKeys = 100000000;\nvar wordCounts = ems.new({\n dimensions: [maxNKeys], // Maximum # of different keys the array can store\n heapSize: maxNKeys * 10, // 10 bytes of storage per key, used for key (dictionary word) itself\n useMap: true, // Use a key-index mapping, not integer indexes\n setFEtags: 'full', // Initial full/empty state of array elements\n doDataFill: true, // Initialize data values\n dataFill: 0 // Initial value of new keys\n});\n\n\n//-------------------------------------------------------------------\n// Use a key-value mapped array to store execution statistics\nvar nStats = 200;\nvar stats = ems.new({\n dimensions: [nStats], // Maximum number of stats that can be stored\n heapSize: nStats * 200,// Space for keys, data values, and sorting\n useMap: true, // Use a key-index mapping, not integer indexes\n setFEtags: 'full', // Initial full/empty state of array elements\n doDataFill: true, // Initialize data values\n dataFill: 0 // Initial value of new keys\n});\n\n\n// ===============================================================================\n// Perform Word Count\n//\n// Use the word as a key to the EMS array, the value is the count of the\n// number of times the word has been encountered. This is incremented\n// atomically.\n//\n// Bookkeeping for statistics is performed by keeping track of the total number\n// of words and bytes processed in the \"stats\" EMS array.\n//\nvar doc_path = '/path/to/your/document/collection/';\nif (!process.argv[3]) {\n console.log(\"usage: wordCount <# processes> /path/to/your/document/collection/\");\n return -1;\n} else {\n doc_path = process.argv[3];\n}\nvar file_list = fs.readdirSync(doc_path);\nvar splitPattern = new RegExp(/[ \\n,\\.\\\\/_\\-\\<\\>:\\;\\!\\@\\#\\$\\%\\&\\*\\(\\)=\\[\\]|\\\"\\'\\{\\}\\?\\—]/);\n\n\n// Loop over the files in parallel, counting words\nvar startTime = Date.now();\nems.parForEach(0, file_list.length, function (fileNum) {\n try {\n var text = fs.readFileSync(doc_path + file_list[fileNum], 'utf8', \"r\");\n var words = text.replace(/[\\n\\r]/g, ' ').toLowerCase().split(splitPattern);\n // Iterate over all the words in the document\n words.forEach(function (word) {\n // Ignore words over 15 characters long as non-English\n if (word.length < 15 && word.length > 0) {\n // Atomically increment the count of times this word was seen\n var count = wordCounts.faa(word, 1);\n }\n });\n\n // Accumulate some statistics: word and data counters, print progress\n var nWords_read = stats.faa('nWords', words.length);\n var nBytes_read = stats.faa('nBytesRead', text.length);\n\n var now = Date.now();\n if (ems.myID === 0 && fileNum % 10 === 0) {\n console.log(\"Average Words/sec=\" + Math.floor((1000 * nWords_read) / (now - startTime)) +\n \" MBytes/sec=\" + Math.floor((100000 * nBytes_read) / ((now - startTime) * (1 << 20)))/100);\n }\n }\n catch (Err) {\n ems.diag(\"This is not a text file:\" + file_list[fileNum]);\n }\n});\n\n\n\n// ===============================================================================\n// Output the Most Frequently Occurring Words\n//\n// First perform many sequential insertion sorts in parallel,\n// then serially perform a merge sort of all the partial lists.\n//\n// Start by printing the total amount of data processed during word counting\nems.master(function() {\n console.log(\"Totals: \", stats.read('nWords'), \" words parsed, \",\n stats.read('nBytesRead'), \"bytes read.\");\n console.log(\"Starting sort....\");\n});\n\n// Divide the array across all the processes, each process keeps track\n// of the \"local_sort_len\" most frequent word it encounters.\nvar local_sort_len = Math.max(10, process.argv[2]);\nvar biggest_counts = new Array(local_sort_len).fill({\"key\": 0, \"count\": 0});\nems.parForEach(0, maxNKeys, function (keyN) {\n if (keyN % (maxNKeys / 10) === 0) { ems.diag(\"Sorting key ID \" + keyN); }\n var key = wordCounts.index2key(keyN);\n if (key) {\n // Perform an insertion sort of the new key into the biggest_counts\n // list, deleting the last (smallest) element to preserve length.\n var keyCount = wordCounts.read(key);\n var idx = local_sort_len - 1;\n while (idx >= 0 && biggest_counts[idx].count < keyCount) {\n idx -= 1;\n }\n var newBiggest = {\"key\": key, \"count\": keyCount};\n if (idx < 0) {\n biggest_counts = [newBiggest].concat(biggest_counts.slice(0, biggest_counts.length - 1));\n } else if (idx < local_sort_len) {\n var left = biggest_counts.slice(0, idx + 1);\n var right = biggest_counts.slice(idx + 1);\n biggest_counts = left.concat([newBiggest].concat(right)).slice(0, -1);\n }\n }\n});\n\n// Concatenate all the partial (one per process) lists into one list\nstats.writeXF('most_frequent', []);\nems.barrier();\nstats.writeEF('most_frequent', stats.readFE('most_frequent').concat(biggest_counts));\nems.barrier();\n\n\n// Sort & print the list of words, only one process is needed\nems.master(function() {\n biggest_counts = stats.readFF('most_frequent');\n // Sort the list by word frequency\n biggest_counts.sort(function (a, b) {\n return b.count - a.count;\n });\n\n // Print only the first \"local_sort_len\" items -- assume the worst case\n // of all the largest counts are discovered by a single process, the next\n // largest after \"local_sort_len\" is no longer on any list.\n console.log(\"Most frequently appearing terms:\");\n for (var index = 0; index < local_sort_len; index += 1) {\n console.log(index + ': ' + biggest_counts[index].key + \" \" + biggest_counts[index].count);\n }\n});\n"
},
{
"path": "Examples/workQ_and_TM.js",
"content": "/*-----------------------------------------------------------------------------+\n | Extended Memory Semantics (EMS) Version 1.0.0 |\n | Synthetic Semantics http://www.synsem.com/ mogill@synsem.com |\n +-----------------------------------------------------------------------------+\n | Copyright (c) 2011-2014, Synthetic Semantics LLC. All rights reserved. |\n | Copyright (c) 2015-2016, Jace A Mogill. All rights reserved. |\n | |\n | Redistribution and use in source and binary forms, with or without |\n | modification, are permitted provided that the following conditions are met: |\n | * Redistributions of source code must retain the above copyright |\n | notice, this list of conditions and the following disclaimer. |\n | * Redistributions in binary form must reproduce the above copyright |\n | notice, this list of conditions and the following disclaimer in the |\n | documentation and/or other materials provided with the distribution. |\n | * Neither the name of the Synthetic Semantics nor the names of its |\n | contributors may be used to endorse or promote products derived |\n | from this software without specific prior written permission. |\n | |\n | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS |\n | \"AS IS\" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT |\n | LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR |\n | A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL SYNTHETIC |\n | SEMANTICS LLC BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, |\n | EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, |\n | PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR |\n | PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF |\n | LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING |\n | NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS |\n | SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. |\n | |\n +-----------------------------------------------------------------------------+\n | Program Description: |\n | |\n | Enqueue the transactions from a parallel loop. |\n | When all the operations have been queued they all begin |\n | processing transactions. |\n | |\n | Unlike the concurrent_Q_and_TM.js example, randomInRange() is called |\n | from every thread, so it is declared from every thread. |\n | |\n +-----------------------------------------------------------------------------*/\n'use strict';\nvar ems = require('ems')(parseInt(process.argv[2]), true, 'fj');\nvar assert = require('assert');\nvar workQ;\nvar totalNops;\nvar checkNops;\nvar arrLen;\nvar heapSize;\nvar nTransactions;\nvar nTables;\nvar maxNops;\nvar tables;\n\n//-------------------------------------------------------------------\n// Timer functions\nfunction timerStart() {\n return new Date().getTime();\n}\n\nfunction timerStop(timer, nOps, label, myID) {\n function fmtNumber(n) {\n var s = ' ' + n.toString().replace(/\\B(?=(\\d{3})+(?!\\d))/g, \",\");\n if (n < 1) return n;\n else {\n return s.substr(s.length - 15, s.length);\n }\n }\n\n var now = new Date().getTime();\n var opsPerSec = (nOps * 1000000) / ((now - timer) * 1000);\n if (typeof myID === undefined || myID === 0) {\n console.log(fmtNumber(nOps) + label + fmtNumber(Math.floor(opsPerSec).toString()) + \" ops/sec\");\n }\n}\n\n\n//---------------------------------------------------------------------------\n// Initialize shared data: global scalars, EMS buffers for statistics\n// and checksums, an EMS array to be used as a work queue, and many\n// tables to perform transactions on.\n//\nfunction initializeSharedData() {\n arrLen = 1000000; // Maximum number of elements the EMS array can contain\n heapSize = 100000; // Amount of persistent memory to reserve for transactions\n nTransactions = 1000000; // Number of transactions to perform in the experiment\n nTables = 6; // Number of EMS arrays to perform transactions across\n maxNops = 5; // Maximum number of EMS arrays to update during a transaction\n tables = [];\n totalNops = ems.new(2);\n checkNops = ems.new(1);\n\n //---------------------------------------------------------------------------\n // The queue of transactions to perform\n // [ table#, index, read-only ]\n workQ = ems.new({\n dimensions: [nTransactions + ems.nThreads],\n heapSize: nTransactions * 200,\n useExisting: false,\n dataFill: 0,\n doDataFill: true,\n setFEtags: 'empty'\n });\n\n\n //---------------------------------------------------------------------------\n // Create all the tables\n for (var tableN = 0; tableN < nTables; tableN++) {\n tables[tableN] = ems.new({\n dimensions: [arrLen],\n heapSize: 0,\n useExisting: false,\n filename: '/tmp/EMS_tm' + tableN,\n dataFill: 0,\n doDataFill: true,\n setFEtags: 'full'\n });\n }\n}\n\n\n//---------------------------------------------------------------------------\n// Create 'nTransactions' many transactions, each having a random number (up\n// to maxNops) randomly chosen values, some of which are read-only.\n// Because tables and elements are chosen at random, it is necessary to\n// remove duplicate elements in a single transaction, to prevent deadlocks.\n//\n// Transactions are pushed onto a shared queue\nfunction generateTransactions() {\n //---------------------------------------------------------------------------\n // Generate a random integer within a range (inclusive) from 'low' to 'high'\n var randomInRange = function (low, high) {\n return ( Math.floor((Math.random() * (high - low)) + low) );\n };\n\n //---------------------------------------------------------------------------\n // Generate operations involving random elements in random EMS arrays\n // and enqueue them on the work queue\n ems.parForEach(0, nTransactions, function (transN) {\n var ops = [];\n var nOps = randomInRange(1, maxNops);\n for (var opN = 0; opN < nOps; opN++) {\n var tableN = randomInRange(0, nTables);\n var idx = randomInRange(0, arrLen);\n if (transN % 2 == 0 || opN % 3 > 0) {\n ops.push([tableN, idx, true]);\n }\n else {\n ops.push([tableN, idx]);\n }\n }\n\n // De-duplicate operations in a transaction which would deadlock\n var indicies = [];\n var uids = [];\n for (var i = 0; i < ops.length; ++i) {\n indicies[i] = i;\n uids[i] = (ops[i][0] * 1000000000000) + ops[i][1];\n }\n var uniq = [];\n for (opN = 0; opN < ops.length; opN++) {\n var isDupl = false;\n for (var checkN = 0; checkN < ops.length; checkN++) {\n if (opN != checkN && uids[opN] == uids[checkN]) {\n isDupl = true;\n break;\n }\n }\n if (!isDupl) {\n uniq.push(ops[opN]);\n }\n }\n workQ.enqueue(uniq);\n });\n\n // After all the work has been enqueued, add DONE semaphores to the\n // end of the queue so they are processed only after all the work\n // has been issued. Each thread enqueues one event and can only\n // consume one before exiting.\n workQ.enqueue(\"DONE\")\n}\n\n\n//------------------------------------------------------------------\n// Consume transactions. If there is nothing on the queue,\n// keep checking until the \"DONE\" message is dequeued.\n//\nfunction performTransactions() {\n var rwNops = 0;\n var readNops = 0;\n\n while (true) {\n var ops = workQ.dequeue();\n if (ops !== undefined) {\n if (ops === \"DONE\") {\n break;\n } else {\n for (var opN = 0; opN < ops.length; opN++) {\n ops[opN][0] = tables[ops[opN][0]];\n }\n var transaction = ems.tmStart(ops);\n ops.forEach(function (op) {\n var tmp = op[0].read(op[1]);\n if (op[2] != true) {\n rwNops++;\n op[0].write(op[1], tmp + 1);\n } else {\n readNops++;\n }\n });\n ems.tmEnd(transaction, true);\n }\n }\n }\n totalNops.faa(0, rwNops);\n totalNops.faa(1, readNops);\n}\n\n\n//------------------------------------------------------------------------\n// Main program entry point\n//\nems.parallel(initializeSharedData);\ntotalNops.writeXF(0, 0);\ntotalNops.writeXF(1, 0);\ncheckNops.writeXF(0, 0);\n\n// Enqueue all the transaction\nvar startTime = timerStart();\nems.parallel(generateTransactions);\ntimerStop(startTime, nTransactions, \" transactions enqueued \", ems.myID);\n\n// Perform all the transactions\nstartTime = timerStart();\nems.parallel(performTransactions);\ntimerStop(startTime, nTransactions, \" transactions performed\", ems.myID);\ntimerStop(startTime, totalNops.readFF(0), \" table updates \", ems.myID);\ntimerStop(startTime, totalNops.readFF(0) + totalNops.readFF(1), \" elements referenced \", ems.myID);\n\n// Validate the results by summing all the updates\nstartTime = timerStart();\nems.parallel(function () {\n ems.parForEach(0, nTables, function (tableN) {\n var localSum = 0;\n for (var idx = 0; idx < arrLen; idx++) {\n localSum += tables[tableN].read(idx);\n }\n checkNops.faa(0, localSum)\n })\n});\ntimerStop(startTime, nTables * arrLen, \" elements checked \", ems.myID);\n\nassert(checkNops.readFF(0) === totalNops.readFF(0),\n \"Error in final sum = \" + checkNops.readFF(0) + \" should be=\" + totalNops.readFF(0));\n\nconsole.log(\"Success: Got correct final sum = \" + checkNops.readFF(0));\n\n// No more work coming -- Exit on all processes\nems.parallel(function () { process.exit(0); });\n"
},
{
"path": "LICENSE",
"content": "/*-----------------------------------------------------------------------------+\n | Extended Memory Semantics (EMS) Version 1.6.1 |\n +-----------------------------------------------------------------------------+\n | Copyright (c) 2011-2014, Synthetic Semantics LLC. All rights reserved. |\n | Copyright (c) 2015-2020, Jace A Mogill. All rights reserved. |\n | |\n | Redistribution and use in source and binary forms, with or without |\n | modification, are permitted provided that the following conditions are met: |\n | * Redistributions of source code must retain the above copyright |\n | notice, this list of conditions and the following disclaimer. |\n | * Redistributions in binary form must reproduce the above copyright |\n | notice, this list of conditions and the following disclaimer in the |\n | documentation and/or other materials provided with the distribution. |\n | * Neither the name of the Synthetic Semantics nor the names of its |\n | contributors may be used to endorse or promote products derived |\n | from this software without specific prior written permission. |\n | |\n | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS |\n | \"AS IS\" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT |\n | LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR |\n | A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL SYNTHETIC |\n | SEMANTICS LLC BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, |\n | EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, |\n | PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR |\n | PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF |\n | LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING |\n | NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS |\n | SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. |\n | |\n +-----------------------------------------------------------------------------*/\n"
},
{
"path": "Makefile",
"content": "#make\n# *-----------------------------------------------------------------------------+\n# | Extended Memory Semantics (EMS) Version 1.6.1 |\n# | http://mogill.com/ jace@mogill.com |\n# +-----------------------------------------------------------------------------+\n# | Copyright (c) 2016-2020, Jace A Mogill. All rights reserved. |\n# | |\n# | Redistribution and use in source and binary forms, with or without |\n# | modification, are permitted provided that the following conditions are met: |\n# | * Redistributions of source code must retain the above copyright |\n# | notice, this list of conditions and the following disclaimer. |\n# | * Redistributions in binary form must reproduce the above copyright |\n# | notice, this list of conditions and the following disclaimer in the |\n# | documentation and/or other materials provided with the distribution. |\n# | * Neither the name of the Synthetic Semantics nor the names of its |\n# | contributors may be used to endorse or promote products derived |\n# | from this software without specific prior written permission. |\n# | |\n# | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS |\n# | \"AS IS\" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT |\n# | LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR |\n# | A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL SYNTHETIC |\n# | SEMANTICS LLC BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, |\n# | EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, |\n# | PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR |\n# | PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF |\n# | LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING |\n# | NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS |\n# | SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. |\n# | |\n# +-----------------------------------------------------------------------------*\nall: py2 py3 node test help_notice\n\nhelp:\n\t@echo \" Extended Memory Semantics -- Build Targets\"\n\t@echo \"===========================================================\"\n\t@echo \" all Build all targets, run all tests\"\n\t@echo \" node Build only Node.js\"\n\t@echo \" py Build both Python 2 and 3\"\n\t@echo \" \"\n\t@echo \" py[2|3] Build only Python2 or 3\"\n\t@echo \" test Run both Node.js and Py tests\"\n\t@echo \" test[_js|_py|_py2|_py3] Run only Node.js, or only Py tests, respectively\"\n\t@echo \" clean Remove all files that can be regenerated\"\n\t@echo \" clean[_js|_py|_py2|_py3] Remove Node.js or Py files that can be regenerated\"\n\nhelp_notice:\n\t@echo \"=== \\\"make help\\\" for list of targets\"\n\ntest: test_js test_py\n\ntest_js: node\n\tnpm test\n\ntest_py: test_py2 test_py3\n\ntest_py3: py3\n\t(cd Tests; python3 ./py_api.py)\n\ntest_py2: py2\n\t(cd Tests; python ./py_api.py)\n\nnode: build/Release/ems.node\n\nbuild/Release/ems.node:\n\t(cd node_modules; /bin/rm -f ems)\n\tnpm install\n\t(cd node_modules; ln -s ../ ./ems)\n\npy: py2 py3\n\npy3:\n\t(cd Python; sudo rm -rf Python/build Python/ems.egg-info Python/dist; sudo python3 ./setup.py build --build-temp=./ install)\n\npy2:\n\t(cd Python; sudo rm -rf Python/build Python/ems.egg-info Python/dist; sudo python ./setup.py build --build-temp=./ install)\n\nclean: clean_js clean_py3 clean_py2\n\nclean_js:\n\t$(RM) -rf build\n\nclean_py3:\n\t$(RM) -rf Python/build Python/py3ems/build /usr/local/lib/python*/dist-packages/*ems* ~/Library/Python/*/lib/python/site-packages/*ems* ~/Library/Python/*/lib/python/site-packages/__pycache__/*ems* /Library/Frameworks/Python.framework/Versions/*/lib/python*/site-packages/*ems*\n\nclean_py2:\n\t$(RM) -rf Python/build Python/py2ems/build /usr/local/lib/python*/dist-packages/*ems* ~/Library/Python/*/lib/python/site-packages/*ems* ~/Library/Python/*/lib/python/site-packages/__pycache__/*ems* /Library/Python/*/site-packages/*ems*\n"
},
{
"path": "Python/README.md",
"content": "# Python 2 and 3\n\nThe FFI differs slightly between Python 2 and 3, these are papered over\nat runtime. There are, of course, the usual \"print\" shenanigans.\nUsing the \"test_py\" make target, both platforms are built and tested.\n\nThe user API does not change between Python 2 and 3.\n"
},
{
"path": "Python/__init__.py",
"content": "from ems import *\n\n"
},
{
"path": "Python/ems.py",
"content": "\"\"\"\n +-----------------------------------------------------------------------------+\n | Extended Memory Semantics (EMS) Version 1.5.0 |\n | Synthetic Semantics http://www.synsem.com/ mogill@synsem.com |\n +-----------------------------------------------------------------------------+\n | Copyright (c) 2016-2017, Jace A Mogill. All rights reserved. |\n | |\n | Redistribution and use in source and binary forms, with or without |\n | modification, are permitted provided that the following conditions are met: |\n | * Redistributions of source code must retain the above copyright |\n | notice, this list of conditions and the following disclaimer. |\n | * Redistributions in binary form must reproduce the above copyright |\n | notice, this list of conditions and the following disclaimer in the |\n | documentation and/or other materials provided with the distribution. |\n | * Neither the name of the Synthetic Semantics nor the names of its |\n | contributors may be used to endorse or promote products derived |\n | from this software without specific prior written permission. |\n | |\n | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS |\n | \"AS IS\" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT |\n | LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR |\n | A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL SYNTHETIC |\n | SEMANTICS LLC BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, |\n | EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, |\n | PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR |\n | PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF |\n | LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING |\n | NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS |\n | SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. |\n | |\n +-----------------------------------------------------------------------------+\n\"\"\"\nimport sys\nimport os\nimport math\nimport json\nimport re\nimport subprocess\nfrom multiprocessing import Process, Pipe\nfrom cffi import FFI\nimport site\n\n\"\"\"\nsudo apt-get install python-pip\nsudo apt-get install libffi-dev\nsudo pip install cffi\nsudo apt-get install python3-cffi\n\"\"\"\n\n\nffi = FFI()\ncpp_out = subprocess.check_output([\"cpp\", \"../src/ems_proto.h\"]).decode(\"utf-8\")\nprototypes = cpp_out.split(\"\\n\")\nheaderLines = []\nfor line in prototypes:\n # Strip CPP directives and annotations\n line = re.sub(\"^#.*$\", \"\", line)\n # Strip extern attribute\n line = re.sub(\"extern \\\"C\\\" \", \"\", line)\n if line is not \"\":\n headerLines.append(line)\n\n# Delcare the CFFI Headers\nffi.cdef('\\n'.join(headerLines))\n\n# Find the .so and load it\nlibems = None\npackage_paths = site.getsitepackages()\nfor package_path in package_paths:\n try:\n packages = os.listdir(package_path)\n for package in packages:\n if package == \"libems\" and libems is None:\n libems_path = package_path + \"/libems/\"\n files = os.listdir(libems_path)\n for file in files:\n if file[-3:] == \".so\": # TODO: Guessing it's the only .so\n fname = libems_path + file\n libems = ffi.dlopen(fname)\n break\n except:\n # print(\"This path does not exist:\", package_path, \"|\", type (package_path))\n pass\n\n# Do not GC the EMS values until deleted\nimport weakref\nglobal_weakkeydict = weakref.WeakKeyDictionary()\n\n# class initialize(object):\n# This enumeration is copied from ems.h\nTYPE_INVALID = 0\nTYPE_BOOLEAN = 1\nTYPE_STRING = 2\nTYPE_FLOAT = 3\nTYPE_INTEGER = 4\nTYPE_UNDEFINED = 5\nTYPE_JSON = 6 # Catch-all for JSON arrays and Objects\n\nTAG_ANY = 4 # Never stored, used for matching\nTAG_RW_LOCK = 3\nTAG_BUSY = 2\nTAG_EMPTY = 1\nTAG_FULL = 0\n\n\ndef emsThreadStub(conn, taskn):\n \"\"\"Function that receives EMS fork-join functions, executes them,\n and returns the results.\"\"\"\n sys.path.append(\"../Python/\")\n global ems\n import ems\n ems.initialize(taskn, True, 'fj', '/tmp/fj_main.ems')\n print(\"STUB: This is the start of it\", taskn)\n ems.myID = taskn\n ems.diag(\"Stub diag. Taskn=\" + str(taskn) + \" myid=\" + str(ems.myID))\n while True:\n msg = conn.recv()\n print(\"CONN RECV ON\", taskn)\n func = msg['func']\n args = msg['args']\n if func is None:\n print(\"Farewell!!!!!!!! from \", taskn)\n conn.close()\n exit(0)\n ems.diag(\"func=\" + str(func) + \" args=\" + str(args))\n # print(\"func=\" + str(func) + \" args=\" + str(args))\n retval = func(*args)\n conn.send(str(retval))\n # time.sleep(taskn * 0.01)\n\n\ndef initialize(nThreadsArg, pinThreadsArg=False, threadingType='bsp',\n contextname='/EMS_MainDomain'):\n \"\"\"EMS object initialization, invoked by the require statement\"\"\"\n nThreadsArg = int(nThreadsArg)\n global myID, libems, EMSmmapID, _regionN, pinThreads, domainName, inParallelContext, tasks, nThreads\n if not nThreadsArg > 0:\n print(\"EMS: Must declare number of nodes to use. Input:\" + str(nThreadsArg))\n return\n\n myID = os.getenv(\"EMS_Subtask\")\n if myID is None:\n myID = 0\n else:\n myID = int(myID)\n\n _regionN = 0\n nThreads = nThreadsArg\n pinThreads = pinThreadsArg\n threadingType = threadingType\n domainName = contextname\n c_contextname = _new_EMSval(contextname)\n c_None = _new_EMSval(None)\n c_contextname_char = ffi.cast('char *', c_contextname)\n # All arguments are defined -- now do the EMS initialization\n EMSmmapID = libems.EMSinitialize(0, 0, False,\n c_contextname_char,\n False, False, # 4-5\n False, 0, # 6-7\n c_None, # 8\n False, TAG_FULL, myID, pinThreads, nThreads, 99)\n\n # The master thread has completed initialization, other threads may now\n # safely execute.\n if threadingType == 'bsp':\n inParallelContext = True\n if myID == 0:\n for taskN in range(1, nThreads):\n py_ver = \"python\"\n if sys.version_info[0] == 3:\n py_ver += \"3\"\n os.system('EMS_Subtask=' + str(taskN) + \" \" + py_ver + ' ./' + (' '.join(sys.argv)) + ' &')\n elif threadingType == 'fj':\n if myID == 0:\n inParallelContext = False\n tasks = []\n for taskN in range(1, nThreads):\n parent_conn, child_conn = Pipe()\n p = Process(target=emsThreadStub, args=(child_conn, taskN), name=\"EMS\" + str(taskN))\n p.start()\n print(\"Started job:\", taskN)\n tasks.append((p, parent_conn, child_conn))\n else:\n inParallelContext = True\n elif threadingType == 'user':\n inParallelContext = False\n else:\n print(\"EMS ERROR: Unknown threading model type:\" + str(threadingType))\n myID = -1\n return\n\n\ndef _loop_chunk():\n global myID, libems, EMSmmapID, _regionN, pinThreads, domainName, inParallelContext, tasks, nThreads\n start = ffi.new(\"int32_t *\", 1)\n end = ffi.new(\"int32_t *\", 1)\n if not libems.EMSloopChunk(EMSmmapID, start, end):\n diag(\"_loop_chunk: ERROR -- Not a valid block of iterations\")\n return False\n return {'start': start[0], 'end': end[0]}\n\n\ndef diag(text):\n \"\"\"Print a message to the console with a prefix indicating which thread is printing\n \"\"\"\n global myID, libems, EMSmmapID, _regionN, pinThreads, domainName, inParallelContext, tasks, nThreads\n print(\"EMStask \" + str(myID) + \": \" + text)\n\n\ndef parallel(func, *kargs):\n \"\"\"Co-Begin a FJ parallel region, executing the function 'func'\"\"\"\n global myID, libems, EMSmmapID, _regionN, pinThreads, domainName, inParallelContext, tasks, nThreads\n # print(\"EMSparallel: starting\", len(tasks), tasks)\n inParallelContext = True\n taskN = 1\n for proc, parent_conn, child_conn in tasks:\n print(\"Sending to\", kargs[0])\n parent_conn.send({'func': func, 'args': kargs})\n taskN += 1\n for proc, parent_conn, child_conn in tasks:\n retval = parent_conn.recv()\n print(\"ParRegion ret: \", retval)\n # assert retval ==\n print(\"Starting local copy in parallel region\")\n func(*kargs) # Perform the work on this process\n print(\"barrier at end of perallel region\")\n barrier()\n inParallelContext = False\n\n\ndef parForEach(start, # First iteration's index\n end, # Final iteration's index\n loopBody, # Function to execute each iteration\n scheduleType = 'guided', # Load balancing technique\n minChunk = 1 # Smallest block of iterations\n ):\n \"\"\"Execute the local iterations of a decomposed loop with\n the specified scheduling.\n \"\"\"\n global myID, libems, EMSmmapID, _regionN, pinThreads, domainName, inParallelContext, tasks, nThreads\n\n if scheduleType == 'static': # Evenly divide the iterations across threads\n blocksz = math.floor((end - start) / nThreads) + 1\n s = int((blocksz * myID) + start)\n e = int((blocksz * (myID + 1)) + start)\n if e > end:\n e = end\n for idx in range(s, e):\n loopBody(idx)\n else: # scheduleType == 'dynamic' or scheduleType == 'guided' or default\n # Initialize loop bounds, block size, etc.\n if scheduleType == 'guided':\n ems_sched_type = 1200 # From ems.h\n elif scheduleType == 'dynamic':\n ems_sched_type = 1201 # From ems.h\n else:\n print(\"EMS parForEach: Invalid scheduling type:\", scheduleType)\n return\n libems.EMSloopInit(EMSmmapID, start, end, minChunk, ems_sched_type)\n # Do not enter loop until all threads have completed initialization\n # If the barrier is present at the loop end, this may replaced\n # with first-thread initialization.\n barrier()\n extents = _loop_chunk()\n while extents['end'] - extents['start'] > 0:\n for idx in range(extents['start'], extents['end']):\n loopBody(idx)\n extents = _loop_chunk()\n\n # Do not proceed until all iterations have completed\n # TODO: OpenMP offers NOWAIT to skip this barrier\n barrier()\n\n\ndef tmStart(emsElems):\n \"\"\"Start a Transaction\n Input is an array containing EMS elements to transition from\n Full to Empty:\n arr = [ [ emsArr0, idx0 ], [ emsArr1, idx1, true ], [ emsArr2, idx2 ] ]\n \"\"\"\n global myID, libems, EMSmmapID, _regionN, pinThreads, domainName, inParallelContext, tasks, nThreads\n\n def getElemUID(item):\n # print(\"item=\", item[0], item[1], item[0].index2key(item[1]) )\n # return (item[0].mmapID * MAX_ELEM_PER_REGION) + item[0].index2key(item[1])\n return str(item[0].mmapID * MAX_ELEM_PER_REGION) + str(item[1])\n\n MAX_ELEM_PER_REGION = 10000000000000\n\n # Build a list of indexes of the elements to lock which will hold\n # the elements sorted for deadlock free acquisition\n # Sort the elements to lock according to a global ordering\n sortedElems = sorted(emsElems, key=getElemUID)\n\n # Acquire the locks in the deadlock free order, saving the contents\n # of the memory when it locked.\n # Mark read-write data as Empty, and read-only data under a readers-writer lock\n tmHandle = []\n for elem in sortedElems:\n if len(elem) > 2:\n val = elem[0].readRW(elem[1])\n readonly = True\n else:\n val = elem[0].readFE(elem[1])\n readonly = False\n tmHandle.append([elem[0], elem[1], readonly, val])\n return tmHandle\n\n\ndef tmEnd(\n tmHandle, # The returned value from tmStart\n doCommit # Commit or Abort the transaction\n ):\n \"\"\"Commit or abort a transaction\n The tmHandle contains the result from tmStart:\n [ [ EMSarray, index, isReadOnly, origValue ], ... ]\n \"\"\"\n global myID, libems, EMSmmapID, _regionN, pinThreads, domainName, inParallelContext, tasks, nThreads\n for emsElem in tmHandle:\n if doCommit:\n if emsElem[2]:\n # Is a read-only element\n emsElem[0].releaseRW(emsElem[1])\n else:\n # Is read-write, keep current value and mark Full\n emsElem[0].setTag(emsElem[1], 'full')\n else:\n # Abort the transaction\n if emsElem[2]:\n # Is read-only element\n emsElem[0].releaseRW(emsElem[1])\n else:\n # Write back the original value and mark full\n emsElem[0].writeEF(emsElem[1], emsElem[3])\n\n\n# -------------------------------------------------\ndef critical(func, timeout=1000000):\n \"\"\"Serialize execution through this function\"\"\"\n global myID, libems, EMSmmapID, _regionN, pinThreads, domainName, inParallelContext, tasks, nThreads\n libems.EMScriticalEnter(EMSmmapID, timeout)\n retObj = func()\n libems.EMScriticalExit(EMSmmapID)\n return retObj\n\n\ndef master(func):\n \"\"\"Perform func only on thread 0\"\"\"\n global myID, libems, EMSmmapID, _regionN, pinThreads, domainName, inParallelContext, tasks, nThreads\n if myID == 0:\n return func()\n\n\ndef single(func):\n \"\"\"Perform the function func once by the first thread to reach\n the function. The final barrier is required because a\n thread may try to execute the next single-execution region\n before other threads have finished this region, which the EMS\n runtime cannot tell apart. Barriers are phased, so a barrier\n is used to prevent any thread from entering the next single-\n execution region before this one is complete\n \"\"\"\n global myID, libems, EMSmmapID, _regionN, pinThreads, domainName, inParallelContext, tasks, nThreads\n retObj = None\n if libems.EMSsingleTask(EMSmmapID):\n retObj = func()\n barrier()\n return retObj\n\n\ndef barrier(timeout=10000):\n \"\"\"Wrapper around the EMS global barrier\"\"\"\n global myID, libems, EMSmmapID, _regionN, pinThreads, domainName, inParallelContext, tasks, nThreads\n if inParallelContext:\n return libems.EMSbarrier(EMSmmapID, timeout)\n else:\n return 1 # Return a positive non-zero integer -- as if time left on clock\n\n\ndef new(arg0=None, # Maximum number of elements the EMS region can hold\n heapSize=0, # #bytes of memory reserved for strings/arrays/objs/maps/etc\n filename=None): # Optional filename for persistent EMS memory\n \"\"\"Creating a new EMS memory region\"\"\"\n global myID, libems, EMSmmapID, _regionN, pinThreads, domainName, inParallelContext, tasks, nThreads\n fillIsJSON = False\n emsDescriptor = EMSarray( # Internal EMS descriptor\n nElements=1, # Required: Maximum number of elements in array\n heapSize=0, # Optional, default=0: Space, in bytes, for strings, maps, objects, etc.\n mlock=0, # Optional, 0-100% of EMS memory into RAM\n useMap=False, # Optional, default=False: Use a map from keys to indexes\n useExisting=False, # Optional, default=false: Preserve data if a file already exists\n persist=True, # Optional, default=true: Preserve the file after threads exit\n doDataFill=False, # Optional, default=false: Data values should be initialized\n dataFill=None, # Optional, default=false: Value to initialize data to\n dimStride=[] # Stride factors for each dimension of multidimensional arrays\n )\n\n emsDescriptor.dataFill = _new_EMSval(None)\n\n if arg0 is None: # Nothing passed in, assume length 1\n emsDescriptor.dimensions = [1]\n else:\n if type(arg0) == dict: # User passed in emsArrayDescriptor\n if 'dimensions' in arg0:\n if type(arg0['dimensions']) == list:\n emsDescriptor.dimensions = arg0['dimensions']\n else:\n emsDescriptor.dimensions = [arg0['dimensions']]\n\n if 'heapSize' in arg0:\n emsDescriptor.heapSize = arg0['heapSize']\n\n if 'mlock' in arg0:\n emsDescriptor.mlock = arg0['mlock']\n\n if 'useMap' in arg0:\n emsDescriptor.useMap = arg0['useMap']\n\n if 'filename' in arg0:\n emsDescriptor.filename = arg0['filename']\n\n if 'persist' in arg0:\n emsDescriptor.persist = arg0['persist']\n\n if 'useExisting' in arg0:\n emsDescriptor.useExisting = arg0['useExisting']\n\n if 'doDataFill' in arg0:\n if arg0['doDataFill']:\n emsDescriptor.doDataFill = True\n emsDescriptor.dataFill = _new_EMSval(arg0['dataFill'])\n\n if 'doSetFEtags' in arg0:\n emsDescriptor.doSetFEtags = arg0['doSetFEtags']\n\n if 'setFEtags' in arg0:\n if (arg0['setFEtags'] == 'full'):\n emsDescriptor.setFEtagsFull = True\n else:\n emsDescriptor.setFEtagsFull = False\n else:\n if type(arg0) == list: # User passed in multi-dimensional array\n emsDescriptor.dimensions = arg0\n else:\n if type(arg0) == int: # User passed in scalar 1-D array length\n emsDescriptor.dimensions = [arg0]\n else:\n print(\"EMSnew: ERROR Non-integer type of arg0\", str(arg0), type(arg0))\n\n if heapSize > 0:\n emsDescriptor.heapSize = heapSize\n\n if emsDescriptor.heapSize <= 0 and emsDescriptor.useMap:\n print(\"WARNING: New EMS array with no heap, disabling mapped keys\")\n emsDescriptor.useMap = False\n\n if type(filename) == str:\n emsDescriptor.filename = filename\n\n if not emsDescriptor.dimensions:\n emsDescriptor.dimensions = [emsDescriptor.nElements]\n\n # Compute the stride factors for each dimension of a multidimensal array\n for dim in emsDescriptor.dimensions:\n emsDescriptor.dimStride.append(emsDescriptor.nElements)\n emsDescriptor.nElements *= dim\n\n # Name the region if a name wasn't given\n if not emsDescriptor.filename:\n emsDescriptor.filename = '/EMS_region_' + str(_regionN)\n emsDescriptor.persist = False\n\n if emsDescriptor.useExisting:\n try:\n fh = open(str(emsDescriptor.filename), 'r')\n fh.close()\n except Exception as err:\n print(\"EMS ERROR: file \" + str(emsDescriptor.filename) + \" should already exist, but does not. \" + str(err))\n return\n\n # init() is first called from thread 0 to perform one-thread\n # only operations (ie: unlinking an old file, opening a new\n # file). After thread 0 has completed initialization, other\n # threads can safely share the EMS array.\n if not emsDescriptor.useExisting and myID != 0:\n barrier()\n\n emsDescriptor.mmapID = libems.EMSinitialize(\n emsDescriptor.nElements,\n emsDescriptor.heapSize, # 1\n emsDescriptor.useMap,\n emsDescriptor.filename.encode(), # 3\n emsDescriptor.persist,\n emsDescriptor.useExisting, # 5\n emsDescriptor.doDataFill,\n fillIsJSON,\n emsDescriptor.dataFill,\n emsDescriptor.doSetFEtags,\n emsDescriptor.setFEtagsFull,\n myID, pinThreads, nThreads,\n emsDescriptor.mlock\n )\n if not emsDescriptor.useExisting and myID == 0:\n barrier()\n\n _regionN += 1\n barrier() # Wait until all processes finished initialization\n return emsDescriptor\n\n'''\n# ================================================\ndef ems_thread_stub(taskN, conn):\n global myID, libems, EMSmmapID, _regionN, pinThreads, domainName, inParallelContext, tasks, nThreads\n myID = taskN\n while True:\n print(\"THread Stub waiting for message\", taskN, conn)\n mesg = conn.recv()\n try:\n print(\"THread Stub message:\", taskN, *mesg)\n mesg['func'](*mesg['args'])\n except:\n print(\"Had exception in the stub\", taskN)\n'''\n\n# =======================================================================================\ndef _new_EMSval(val):\n global myID, libems, EMSmmapID, _regionN, pinThreads, domainName, inParallelContext, tasks, nThreads\n global global_weakkeydict\n emsval = ffi.new(\"EMSvalueType *\")\n emsval[0].length = 0\n if type(val) == str:\n if sys.version_info[0] == 2: # Python 2 or 3\n newval = ffi.new('char []', bytes(val))\n else:\n newval = ffi.new('char []', bytes(val, 'utf-8'))\n emsval[0].value = newval\n emsval[0].length = len(newval) + 1\n emsval[0].type = TYPE_STRING\n global_weakkeydict[emsval] = (emsval[0].length, emsval[0].type, emsval[0].value, newval)\n elif type(val) == int:\n emsval[0].type = TYPE_INTEGER\n emsval[0].value = ffi.cast('void *', val)\n elif type(val) == float:\n emsval[0].type = TYPE_FLOAT\n ud_tmp = ffi.new('EMSulong_double *')\n ud_tmp[0].d = ffi.cast('double', val)\n emsval[0].value = ffi.cast('void *', ud_tmp[0].u64)\n elif type(val) == bool:\n emsval[0].type = TYPE_BOOLEAN\n emsval[0].value = ffi.cast('void *', val)\n elif type(val) == list:\n if sys.version_info[0] == 2: # Python 2 or 3\n newval = ffi.new('char []', bytes(json.dumps(val)))\n else:\n newval = ffi.new('char []', bytes(json.dumps(val), 'utf-8'))\n emsval[0].value = newval\n emsval[0].length = len(newval) + 1\n emsval[0].type = TYPE_JSON\n global_weakkeydict[emsval] = (emsval[0].length, emsval[0].type, emsval[0].value, newval)\n elif type(val) == dict:\n if sys.version_info[0] == 2: # Python 2 or 3\n newval = ffi.new('char []', bytes(json.dumps(val)))\n else:\n newval = ffi.new('char []', bytes(json.dumps(val), 'utf-8'))\n emsval[0].value = newval\n emsval[0].length = len(newval) + 1\n emsval[0].type = TYPE_JSON\n global_weakkeydict[emsval] = (emsval[0].length, emsval[0].type, emsval[0].value, newval)\n elif val is None:\n emsval[0].type = TYPE_UNDEFINED\n emsval[0].value = ffi.cast('void *', 0xdeadbeef)\n else:\n print(\"EMS ERROR - unknown type of value:\", type(val), val)\n return None\n return emsval\n\n\n# ==========================================================================================\n\n\nclass EMSarray(object):\n def __init__(self,\n nElements=1, # Required: Maximum number of elements in array\n heapSize=0, # Optional, default=0: Space, in bytes, for strings, maps, objects, etc.\n mlock=0, # Optional, 0-100% of EMS memory into RAM\n useMap=False, # Optional, default=false: Use a map from keys to indexes\n useExisting=False, # Optional, default=false: Preserve data if a file already exists\n persist=True, # Optional, default=true: Preserve the file after threads exit\n doDataFill=False, # Optional, default=false: Data values should be initialized\n dataFill=None, # Optional, default=false: Value to initialize data to\n dimStride=[] # Stride factors for each dimension of multidimensional arrays\n ):\n global myID, libems, EMSmmapID, _regionN, pinThreads, domainName, inParallelContext, tasks, nThreads\n # self.ems = ems\n self.nElements = nElements\n self.heapSize = heapSize\n self.mlock = mlock\n self.useMap = useMap\n self.useExisting = useExisting\n self.persist = persist\n self.doSetFEtags = True\n self.setFEtags = 'full'\n self.doDataFill = doDataFill\n self.dataFill = dataFill\n self.dimStride = dimStride\n self.dimensions = None\n self.filename = None\n self.mmapID = -1\n self.mlock = 1\n self.doSetFEtags = False # Optional, initialize full/empty tags\n self.setFEtagsFull = True # Optional, used only if doSetFEtags is true\n\n # set any attributes here - before initialisation\n # these remain as normal attributes\n # self.attribute = attribute\n # dict.__init__(self, {})\n self.__initialised = True\n # after initialisation, setting attributes is the same as setting an item\n\n def destroy(self, unlink_file):\n \"\"\"Release all resources associated with an EMS memory region\"\"\"\n global myID, libems, EMSmmapID, _regionN, pinThreads, domainName, inParallelContext, tasks, nThreads\n barrier()\n if myID == 0:\n libems.EMSdestroy(self.mmapID, unlink_file)\n barrier()\n\n def _returnData(self, emsval):\n global myID, libems, EMSmmapID, _regionN, pinThreads, domainName, inParallelContext, tasks, nThreads\n if emsval[0].type == TYPE_STRING:\n if sys.version_info[0] == 2: # Python 2 or 3\n return ffi.string(ffi.cast(\"char *\", emsval[0].value))\n else:\n return ffi.string(ffi.cast('char *', emsval[0].value)).decode('utf-8')\n elif emsval[0].type == TYPE_JSON:\n if sys.version_info[0] == 2: # Python 2 or 3\n json_str = ffi.string(ffi.cast(\"char *\", emsval[0].value))\n else:\n json_str = ffi.string(ffi.cast('char *', emsval[0].value)).decode('utf-8')\n tmp_str = \"{\\\"data\\\":\" + json_str + \"}\"\n return json.loads(tmp_str)['data']\n elif emsval[0].type == TYPE_INTEGER:\n return int(ffi.cast('int64_t', emsval[0].value))\n elif emsval[0].type == TYPE_FLOAT:\n ud_tmp = ffi.new('EMSulong_double *')\n ud_tmp[0].u64 = ffi.cast('uint64_t', emsval[0].value)\n return ud_tmp[0].d\n elif emsval[0].type == TYPE_BOOLEAN:\n return bool(int(ffi.cast('uint64_t', emsval[0].value)))\n elif emsval[0].type == TYPE_UNDEFINED:\n return None\n else:\n print(\"EMS ERROR - unknown type of value:\", type(emsval), emsval)\n return None\n\n def sync(self):\n \"\"\"Synchronize memory with storage\"\"\"\n global myID, libems, EMSmmapID, _regionN, pinThreads, domainName, inParallelContext, tasks, nThreads\n return libems.EMSsync(self.mmapID)\n\n def index2key(self, index):\n global myID, libems, EMSmmapID, _regionN, pinThreads, domainName, inParallelContext, tasks, nThreads\n key = _new_EMSval(None)\n assert libems.EMSindex2key(self.mmapID, index, key)\n return self._returnData(key)\n\n # ==================================================================\n # Wrappers around Stacks and Queues\n def push(self, value):\n nativeValue = _new_EMSval(value)\n return libems.EMSpush(self.mmapID, nativeValue)\n\n def pop(self):\n val = _new_EMSval(None)\n libems.EMSpop(self.mmapID, val)\n return self._returnData(val)\n\n def dequeue(self):\n val = _new_EMSval(None)\n libems.EMSdequeue(self.mmapID, val)\n return self._returnData(val)\n\n def enqueue(self, value):\n nativeValue = _new_EMSval(value)\n return libems.EMSenqueue(self.mmapID, nativeValue)\n\n # ==================================================================\n # Wrappers around Primitive AMOs\n # Translate EMS maps and multi-dimensional array indexes/keys\n # into EMS linear addresses\n # Apparently it is illegal to pass a native function as an argument\n def write(self, indexes, value):\n nativeIndex = _new_EMSval(self._idx(indexes))\n nativeValue = _new_EMSval(value)\n libems.EMSwrite(self.mmapID, nativeIndex, nativeValue)\n return (self.mmapID, nativeIndex, nativeValue)\n\n def writeEF(self, indexes, value):\n nativeIndex = _new_EMSval(self._idx(indexes))\n nativeValue = _new_EMSval(value)\n libems.EMSwriteEF(self.mmapID, nativeIndex, nativeValue)\n return (self.mmapID, nativeIndex, nativeValue)\n\n def writeXF(self, indexes, value):\n nativeIndex = _new_EMSval(self._idx(indexes))\n nativeValue = _new_EMSval(value)\n libems.EMSwriteXF(self.mmapID, nativeIndex, nativeValue)\n return (self.mmapID, nativeIndex, nativeValue)\n\n def writeXE(self, indexes, value):\n nativeIndex = _new_EMSval(self._idx(indexes))\n nativeValue = _new_EMSval(value)\n libems.EMSwriteXE(self.mmapID, nativeIndex, nativeValue)\n return (self.mmapID, nativeIndex, nativeValue)\n\n # ---------------------------------------------------\n def read(self, indexes):\n emsnativeidx = _new_EMSval(self._idx(indexes))\n val = _new_EMSval(None)\n libems.EMSread(self.mmapID, emsnativeidx, val)\n return self._returnData(val)\n\n def readFE(self, indexes):\n emsnativeidx = _new_EMSval(self._idx(indexes))\n val = _new_EMSval(None)\n libems.EMSreadFE(self.mmapID, emsnativeidx, val)\n return self._returnData(val)\n\n def readFF(self, indexes):\n emsnativeidx = _new_EMSval(self._idx(indexes))\n val = _new_EMSval(None)\n libems.EMSreadFF(self.mmapID, emsnativeidx, val)\n return self._returnData(val)\n\n def readRW(self, indexes):\n emsnativeidx = _new_EMSval(self._idx(indexes))\n val = _new_EMSval(None)\n libems.EMSreadRW(self.mmapID, emsnativeidx, val)\n return self._returnData(val)\n\n def releaseRW(self, indexes):\n emsnativeidx = _new_EMSval(self._idx(indexes))\n return libems.EMSreleaseRW(self.mmapID, emsnativeidx)\n\n def setTag(self, indexes, fe):\n emsnativeidx = _new_EMSval(self._idx(indexes))\n return libems.EMSsetTag(self.mmapID, emsnativeidx, (fe == 'full'))\n\n # ---------------------------------------------------\n # Atomic RMW\n def faa(self, indexes, val):\n if type(val) == dict:\n print(\"EMSfaa ERROR: Cannot add an object to something\")\n return val\n else:\n ems_nativeidx = _new_EMSval(self._idx(indexes))\n ems_val = _new_EMSval(val)\n ems_retval = _new_EMSval(None)\n assert libems.EMSfaa(self.mmapID, ems_nativeidx, ems_val, ems_retval)\n return self._returnData(ems_retval)\n\n def cas(self, indexes, oldVal, newVal):\n if type(oldVal) == dict:\n print(\"EMScas ERROR: Cannot compare objects, only JSON primitives\")\n return None\n else:\n ems_nativeidx = _new_EMSval(self._idx(indexes))\n ems_oldval = _new_EMSval(oldVal)\n ems_newval = _new_EMSval(newVal)\n ems_retval = _new_EMSval(None)\n libems.EMScas(self.mmapID, ems_nativeidx, ems_oldval, ems_newval, ems_retval)\n return self._returnData(ems_retval)\n\n def _idx(self, indexes):\n idx = 0\n if type(indexes) == list: # Is a Multidimension array: [x,y,z]\n rank = 0\n for index in indexes:\n idx += index * self.dimStride[rank]\n rank += 1\n else:\n if not type(indexes) == int and not self.useMap: # If no map, only use integers\n print(\"EMS ERROR: Non-integer index used, but EMS memory was not configured to use a map (useMap)\",\n indexes, type(indexes), self.useMap)\n idx = -1\n else: # Is a mappable intrinsic type\n idx = indexes\n return idx\n\n def __getattr__(self, attr):\n return self.read(attr)\n\n def __setattr__(self, attr, value):\n if not '_EMSarray__initialised' in self.__dict__ or attr in self.__dict__:\n # Ignore object initialization and normal attribute access\n return dict.__setattr__(self, attr, value)\n else:\n return self.write(attr, value)\n\n def __getitem__(self, item):\n return EMSelement(self, item)\n\n def __setitem__(self, key, value):\n return self.write(key, value)\n\n\n# =============================================================================================\n\nclass EMSelement(object):\n def __init__(self, ems_array, index):\n self._ems_array = ems_array\n self._index = index\n\n def write(self, value):\n return self._ems_array.write(self._index, value)\n\n def writeXE(self, value):\n return self._ems_array.writeXE(self._index, value)\n\n def writeXF(self, value):\n return self._ems_array.writeXF(self._index, value)\n\n def writeEF(self, value):\n return self._ems_array.writeEF(self._index, value)\n\n def read(self):\n return self._ems_array.read(self._index)\n\n def readFF(self):\n return self._ems_array.readFF(self._index)\n\n def readFE(self):\n return self._ems_array.readFE(self._index)\n\n def releaseRW(self):\n return self._ems_array.releaseRW(self._index)\n\n def readRW(self):\n return self._ems_array.readRW(self._index)\n\n def setTag(self, fe):\n return self._ems_array.setTag(self._index, fe)\n\n def faa(self, value):\n return self._ems_array.faa(self._index, value)\n\n def cas(self, oldVal, newVal):\n return self._ems_array.cas(self._index, oldVal, newVal)\n"
},
{
"path": "Python/setup.py",
"content": "\"\"\"\n +-----------------------------------------------------------------------------+\n | Extended Memory Semantics (EMS) Version 1.6.1 |\n | Synthetic Semantics http://www.synsem.com/ mogill@synsem.com |\n +-----------------------------------------------------------------------------+\n | Copyright (c) 2016, Jace A Mogill. All rights reserved. |\n | |\n | Redistribution and use in source and binary forms, with or without |\n | modification, are permitted provided that the following conditions are met: |\n | * Redistributions of source code must retain the above copyright |\n | notice, this list of conditions and the following disclaimer. |\n | * Redistributions in binary form must reproduce the above copyright |\n | notice, this list of conditions and the following disclaimer in the |\n | documentation and/or other materials provided with the distribution. |\n | * Neither the name of the Synthetic Semantics nor the names of its |\n | contributors may be used to endorse or promote products derived |\n | from this software without specific prior written permission. |\n | |\n | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS |\n | \"AS IS\" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT |\n | LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR |\n | A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL SYNTHETIC |\n | SEMANTICS LLC BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, |\n | EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, |\n | PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR |\n | PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF |\n | LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING |\n | NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS |\n | SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. |\n | |\n +-----------------------------------------------------------------------------+\n\"\"\"\nfrom distutils.core import setup, Extension\nimport os\nimport sys\n\n# Path to C library source code\nsrc_path = '../src/'\n\n# OS Specific link flags\nlink_args = []\nif sys.platform == \"linux\" or sys.platform == \"linux2\":\n link_args.append(\"-lrt\")\nelif sys.platform == \"darwin\":\n pass\nelse:\n pass\n\nsetup(\n name=\"ems\",\n version=\"1.6.1\",\n py_modules=[\"ems\"],\n setup_requires=[\"cffi>=1.0.0\"],\n install_requires=[\"cffi>=1.0.0\"],\n\n # Author details\n author='Jace A Mogill',\n author_email='jace@mogill.com',\n\n description='Extended Memory Semantics (EMS) for Python',\n license='BSD',\n\n # The project's main homepage.\n url='https://github.com/mogill/ems',\n\n ext_modules=[Extension('libems.so',\n [src_path + filename for filename in\n ['collectives.cc', 'ems.cc', 'ems_alloc.cc', 'loops.cc', 'primitives.cc', 'rmw.cc']],\n extra_link_args=link_args\n )],\n long_description='Persistent Shared Memory and Parallel Programming Model',\n keywords=[\"non volatile memory\",\n \"NVM\",\n \"NVMe\",\n \"multithreading\",\n \"multithreaded\",\n \"parallel\",\n \"parallelism\",\n \"concurrency\",\n \"shared memory\",\n \"multicore\",\n \"manycore\",\n \"transactional memory\",\n \"TM\",\n \"persistent memory\",\n \"pmem\",\n \"Extended Memory Semantics\",\n \"EMS\"]\n)\n"
},
{
"path": "README.md",
"content": "OSX | Linux | Node 4.1-14.x, Python2/3:\n[](https://travis-ci.org/mogill/ems)\n[](https://www.npmjs.com/package/ems)\n\n### [API Documentation](http://mogill.github.io/ems/Docs/reference.html) | [EMS Website](http://mogill.github.io/ems/Docs/index.html)\n\n\n\n# Extended Memory Semantics (EMS)\n__EMS makes possible persistent shared memory parallelism between Node.js, Python, and C/C++__.\n\nExtended Memory Semantics (EMS) unifies synchronization and storage primitives\nto address several challenges of parallel programming:\n+ Allows any number or kind of processes to share objects\n+ Manages synchronization and object coherency\n+ Implements persistence to non-volatile memory and secondary storage\n+ Provides dynamic load-balancing between processes\n+ May substitute or complement other forms of parallelism\n\n## [Examples: Parallel web servers, word counting](https://github.com/mogill/ems/tree/master/Examples)\n\n#### Table of Contents\n* [Parallel Execution Models Supported](#Types-of-Concurrency) Fork Join, Bulk Synchronous Parallel, User defined\n* [Atomic Operations](#Built-in-Atomic-Operations) Atomic Read-Modify-Write operations\n* [Examples](https://github.com/mogill/ems/tree/master/Examples) Parallel web servers, word counting\n* [Benchmarks](#Examples-and-Benchmarks) Bandwidth, Transaction processing\n* [Synchronization as a Property of the Data, Not a Duty for Tasks](#Synchronization-Property) Full/Empty tags\n* [Installation](#Installation) Downloading from Git or NPM\n* [Roadmap](#Roadmap) The Future™! It's all already happened\n\n#### EMS is targeted at tasks too large for one core or one process but too small for a scalable cluster\n\nA modern multi-core server has 16-32 cores and nearly 1TB of memory,\nequivalent to an entire rack of systems from a few years ago.\nAs a consequence, jobs formerly requiring a Map-Reduce cluster\ncan now be performed entirely in shared memory on a single server\nwithout using distributed programming.\n\n## Sharing Persistent Objects Between Python and Javascript\n\n\nInter-language example in [interlanguage.{js,py}](https://github.com/mogill/ems/tree/master/Examples/Interlanguage)\nThe animated GIF demonstrates the following steps:\n* Start Node.js REPL, create an EMS memory\n* Store \"Hello\"\n* Open a second session, begin the Python REPL\n* Connect Python to the EMS shared memory\n* Show the object created by JS is present in Python\n* Modify the object, and show the modification can be seen in JS\n* Exit both REPLs so no programs are running to \"own\" the EMS memory\n* Restart Python, show the memory is still present\n* Initialize a counter from Python\n* Demonstrate atomic Fetch and Add in JS\n* Start a loop in Python incrementing the counter\n* Simultaneously print and modify the value from JS\n* Try to read \"empty\" data from Python, the process blocks\n* Write the empty memory, marking it full, Python resumes execution\n\n## Types of Concurrency\n
\n
\n
\nEMS extends application capabilities to include transactional memory and\nother fine-grained synchronization capabilities.\n
\nEMS implements several different parallel execution models:\n
\n
Fork-Join Multiprocess: execution begins with a single process that creates new processes\n when needed, those processes then wait for each other to complete.\n\n
Bulk Synchronous Parallel: execution begins with each process starting the program at the\n main entry point and executing all the statements\n\n
User Defined: parallelism may include ad-hoc processes and mixed-language applications\n
\n\t\t
\n
\n
\n \t\t \n
\n
\n
\n
\n
\n
\n \t\t \n
\n
\n
\n
\n \t\t \n
\n
\n
\n
\n\n\n## Built in Atomic Operations\nEMS operations may performed using any JSON data type, read-modify-write operations\nmay use any combination of JSON data types.\nlike operations on ordinary data.\n\nAtomic read-modify-write operations are available\nin all concurrency modes, however collectives are not\navailable in user defined modes.\n\n- __Atomic Operations__:\n\tRead, write, readers-writer lock, read when full and atomically mark empty, write when empty and atomically mark full\n\n- __Primitives__:\n\tStacks, queues, transactions\n\n- __Read-Modify-Write__:\n\tFetch-and-Add, Compare and Swap\n\n- __Collective Operations__:\n\tAll basic [OpenMP](https://en.wikipedia.org/wiki/OpenMP)\n collective operations are implemented in EMS:\n dynamic, block, guided, as are the full complement of static loop scheduling,\n barriers, master and single execution regions\n\n## Examples and Benchmarks\n\n### [API Documentation](http://mogill.github.io/ems/Docs/reference.html) | [EMS Website](http://mogill.github.io/ems/Docs/index.html)\n\n### Word Counting Using Atomic Operations\n[Word counting example](https://github.com/mogill/ems/tree/master/Examples)\n\nMap-Reduce is often demonstrated using word counting because each document can\nbe processed in parallel, and the results of each document's dictionary reduced\ninto a single dictionary. This EMS implementation also\niterates over documents in parallel, but it maintains a single shared dictionary\nacross processes, atomically incrementing the count of each word found.\nThe final word counts are sorted and the most frequently appearing words\nare printed with their counts.\n\n\n\nThe performance of this program was measured using an Amazon EC2 instance: \n`c4.8xlarge (132 ECUs, 36 vCPUs, 2.9 GHz, Intel Xeon E5-2666v3, 60 GiB memory`\nThe leveling of scaling around 16 cores despite the presence of ample work\nmay be related to the use of non-dedicated hardware:\nHalf of the 36 vCPUs are presumably HyperThreads or otherwise shared resource.\nAWS instances are also bandwidth limited to EBS storage, where our Gutenberg\ncorpus is stored.\n\n### Bandwidth Benchmarking\n[STREAMS Example](https://github.com/mogill/ems/tree/master/Examples/STREAMS)\n\nA benchmark similar to [STREAMS](https://www.cs.virginia.edu/stream/)\ngives us the maximum speed EMS double precision\nfloating point operations can be performed on a\n`c4.8xlarge (132 ECUs, 36 vCPUs, 2.9 GHz, Intel Xeon E5-2666v3, 60 GiB memory`.\n\n\n\n\n### Benchmarking of Transactions and Work Queues\n[Transactions and Work Queues Example](https://github.com/mogill/ems/tree/master/Examples)\n\nTransactional performance is measured alone, and again with a separate\nprocess appending new processes as work is removed from the queue.\nThe experiments were run using an Amazon EC2 instance: \nc4.8xlarge (132 ECUs, 36 vCPUs, 2.9 GHz, Intel Xeon E5-2666v3, 60 GiB memory\n\n#### Experiment Design\nSix EMS arrays are created, each holding 1,000,000 numbers. During the\nbenchmark, 1,000,000 transactions are performed, each transaction involves 1-5\nrandomly selected elements of randomly selected EMS arrays.\nThe transaction reads all the elements and\nperforms a read-modify-write operation involving at least 80% of the elements.\nAfter all the transactions are complete, the array elements are checked\nto confirm all the operations have occurred.\n\nThe parallel process scheduling model used is *block dynamic* (the default),\nwhere each process is responsible for successively smaller blocks\nof iterations. The execution model is *bulk synchronous parallel*, each\nprocesses enters the program at the same main entry point\nand executes all the statements in the program.\n`forEach` loops have their normal semantics of performing all iterations,\n`parForEach` loops are distributed across threads, each process executing\nonly a portion of the total iteration space.\n\n\n
\n\t
\n \t
\n\t
\n\t\t\t\n Immediate Transactions: Each process generates a transaction on integer data then immediately performs it.\n \t
\n\t
\n \t
\n\t
\n\t\t\t\n Transactions from a Queue: One of the processes generates the individual transactions and appends\n\t\t\t\tthem to a work queue the other threads get work from.\n Note: As the number of processes increases, the process generating the transactions\n\t\t \tand appending them to the work queue is starved out by processes performing transactions,\n naturally maximizing the data access rate.\n\t
\n\t
\n
\n\t
\n \t
\n\t
\n\t\t\t\n Immediate Transactions on Strings: Each process generates a transaction appending to\n\t\t\ta string, and then immediately performs the transaction.\n \t
\n\t
\n \t
\n
\n Measurements\n
\n Elem. Ref'd: Total number of elements read and/or written\n\t\t \n Table Updates: Number of different EMS arrays (tables) written to\n\t\t \n Trans. Performed: Number of transactions performed across all EMS arrays (tables)\n\t\t \n Trans. Enqueued: Rate transactions are added to the work queue (only 1 generator thread in these experiments)\n\t
\n
\n
\n\n\n\n## [Synchronization as a Property of the Data, Not a Duty for Tasks](#Synchronization-Property)\n\n### [API Documentation](http://mogill.github.io/ems/Docs/reference.html) | [EMS Website](http://mogill.github.io/ems/Docs/index.html)\n\nEMS internally stores tags that are used for synchronization of\nuser data, allowing synchronization to happen independently of\nthe number or kind of processes accessing the data. The tags\ncan be thought of as being in one of three states, _Empty,\nFull,_ or _Read-Only_, and the EMS intrinsic functions\nenforce atomic access through automatic state transitions.\n\nThe EMS array may be indexed directly using an integer, or using a key-index\nmapping from any primitive type. When a map is used, the key and data\nitself are updated atomically.\n\n
\n
\n
\n
\n \n
\n EMS memory is an array of JSON values\n (Number, Boolean, String, Undefined, or Object) accessed using atomic\n operators and/or transactional memory. Safe parallel access\n is managed by passing through multiple gates: First mapping a\n key to an index, then accessing user data protected by EMS\n tags, and completing the whole operation atomically.\n \n
\n
\n
\n
\n \n
EMS Data Tag Transitions & Atomic operations:\n F=Full, E=Empty, X=Don't Care, RW=Readers-Writer lock (# of current readers)\n CAS=Compare-and-Swap, FAA=Fetch-and-Add\n
\n
\n
\n
\n\n\n## More Technical Information\n\nFor a more complete description of the principles of operation,\ncontact the author at ems@rotang.com\n\n[ Complete API reference ](https://github.com/mogill/ems/tree/master/Docs/reference.html)\n\n \n
\n \n
\n\n\n## Installation\n\nBecause all systems are already multicore,\nparallel programs require no additional equipment, system permissions,\nor application services, making it easy to get started.\nThe reduced complexity of\nlightweight threads communicating through shared memory\nis reflected in a rapid code-debug cycle for ad-hoc application development.\n\n### Quick Start with the Makefile\nTo build and test all C, Python 2 and 3, and Node.js targets,\na makefile can automate most build and test tasks.\n\n```sh\ndunlin> make help\n Extended Memory Semantics -- Build Targets\n===========================================================\n all Build all targets, run all tests\n node Build only Node.js\n py Build both Python 2 and 3\n\n py[2|3] Build only Python2 or 3\n test Run both Node.js and Py tests\n test[_js|_py|_py2|_py3] Run only Node.js, or only Py tests, respectively\n clean Remove all files that can be regenerated\n clean[_js|_py|_py2|_py3] Remove Node.js or Py files that can be regenerated\n```\n\n\n### Install via npm\nEMS is available as a NPM Package. EMS depends on the Node addon API\n(N-API) package.\n\n```sh\nnpm install ems\n```\n\n### Install via GitHub\nDownload the source code, then compile the native code:\n\n```sh\ngit clone https://github.com/mogill/ems.git\ncd ems\nnpm install\n```\n\n\n### Installing for Python\nPython users should download and install EMS git (see above).\nThere is no PIP package, but not due lack of desire or effort.\nA pull request is most welcome!\n\n\n### Run Some Examples\n\nClick here for __[Detailed Examples](https://github.com/mogill/ems/tree/master/Examples)__.\n\nOn a Mac and most Linux\ndistributions EMS will \"just work\", but\nsome Linux distributions restrict access to shared memory. The\nquick workaround is to run jobs as root, a long-term solution will\nvary with Linux distribution.\n\n\nRun the work queue driven transaction processing example on 8 processes:\n```sh\nnpm run \n```\n\nOr manually via:\n```sh\ncd Examples\nnode concurrent_Q_and_TM.js 8\n```\n\nRunning all the tests with 8 processes:\n```sh\nnpm run test # Alternatively: npm test\n```\n\n```sh\ncd Tests\nrm -f EMSthreadStub.js # Do not run the machine generated script used by EMS\nfor test in `ls *js`; do node $test 8; done\n```\n\n## Platforms Supported\nAs of 2016-05-01, Mac/Darwin and Linux are supported. A windows port pull request is welcomed!\n\n\n## Roadmap\nEMS 1.0 uses Nan for long-term Node.js support, we continue to\ndevelop on OSX and Linux via Vagrant.\n\nEMS 1.3 introduces a C API.\n\nEMS 1.4 Python API\n\nEMS 1.4.8 Improved examples and documentation\n\nEMS 1.5 Refactored JS-EMS object conversion temporary storage\n\nEMS 1.6 **[This Release]** Updated to replace deprecated NodeJS NAN API with the N-API.\n\nEMS 1.7 **[Planned]** Key deletion that frees all resources. Replace open hashing with chaining.\n\nEMS 1.8 **[Planned]** Memory allocator based on\n*R. Marotta, M. Ianni, A. Scarselli, A. Pellegrini and F. Quaglia, \"NBBS: A Non-Blocking Buddy System for Multi-core Machines,\" 2019 19th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGRID), Larnaca, Cyprus, 2019, pp. 11-20, doi: 10.1109/CCGRID.2019.00011.*\n\nEMS 1.9 **[Planned]** Vectorized JSON indexer.\n\nEMS 1.10 **[Planned]** Support for [persistent main system memory (PMEM)](http://pmem.io/) when\nmultiple processes are supported.\n\nEMS 2.0 **[Planned]** New API which more tightly integrates with\nES6, Python, and other dynamically typed languages languages,\nmaking atomic operations on persistent memory more transparent.\n\n## License\nBSD, other commercial and open source licenses are available.\n\n## Links\n[API Documentation](http://mogill.github.io/ems/Docs/reference.html)\n\n[EMS Website](http://mogill.github.io/ems/Docs/index.html)\n\n[Download the NPM](https://www.npmjs.org/package/ems)\n\n[Get the source at GitHub](https://github.com/mogill/ems)\n\n## About\nJace A Mogill specializes in hardware/software co-design of\nresource constrained computing at both the largest and smallest scales.\nHe has over 20 years experience with distributed, multi-core, FPGA, CGRA, GPU, CPU,\nand custom computer architectures.\n\n###### Copyright (C)2011-2020 Jace A Mogill\n"
},
{
"path": "Tests/3dSpace.js",
"content": "/*-----------------------------------------------------------------------------+\n | Extended Memory Semantics (EMS) Version 1.4.0 |\n | Synthetic Semantics http://www.synsem.com/ mogill@synsem.com |\n +-----------------------------------------------------------------------------+\n | Copyright (c) 2011-2014, Synthetic Semantics LLC. All rights reserved. |\n | Copyright (c) 2015-2016, Jace A Mogill. All rights reserved. |\n | |\n | Redistribution and use in source and binary forms, with or without |\n | modification, are permitted provided that the following conditions are met: |\n | * Redistributions of source code must retain the above copyright |\n | notice, this list of conditions and the following disclaimer. |\n | * Redistributions in binary form must reproduce the above copyright |\n | notice, this list of conditions and the following disclaimer in the |\n | documentation and/or other materials provided with the distribution. |\n | * Neither the name of the Synthetic Semantics nor the names of its |\n | contributors may be used to endorse or promote products derived |\n | from this software without specific prior written permission. |\n | |\n | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS |\n | \"AS IS\" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT |\n | LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR |\n | A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL SYNTHETIC |\n | SEMANTICS LLC BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, |\n | EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, |\n | PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR |\n | PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF |\n | LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING |\n | NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS |\n | SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. |\n | |\n +-----------------------------------------------------------------------------*/\nvar ems = require('ems')(parseInt(process.argv[2]), false);\nvar assert = require('assert');\nvar start;\nvar dim1 = ems.new(1000);\nvar dims3d = [800, 120, 50];\nvar dims2d = [5000, 500];\nvar dim3 = ems.new(dims3d, 0, \"/tmp/EMS_3d_space\");\nvar dim2 = ems.new(dims2d, 800000000);\nvar idx = ems.myID;\nvar i, j, k;\n\ndim1.write(idx * 10, idx * 100);\nvar v = dim1.read(idx * 10);\nassert(v == ems.myID * 100, \"Since 1d write-read failed\");\n\nfunction val3d(i, j, k) {\n return i + (j * 10000) + (k * 100000);\n}\n\n\n//-------------------------------------------------------------------\n// Timer function\nfunction stopTimer(timer, nOps, label) {\n function fmtNumber(n) {\n var s = ' ' + n.toString().replace(/\\B(?=(\\d{3})+(?!\\d))/g, \",\");\n if (n < 1) return n;\n else {\n return s.substr(s.length - 15, s.length);\n }\n }\n\n ems.master(function () {\n var now = new Date().getTime();\n var x = (nOps * 1000000) / ((now - timer) * 1000);\n ems.diag(fmtNumber(nOps) + label + fmtNumber(Math.floor(x).toString()) + \" ops/sec\");\n })\n}\n\n//------------------------------------------------------------------------------\n// Spinup loop to physically allocate all memory as RW\nems.barrier();\nstart = new Date().getTime();\nfor (k = ems.myID; k < dims3d[2]; k += ems.nThreads) {\n for (j = 0; j < dims3d[1]; j += 1) {\n for (i = 0; i < dims3d[0]; i += 1) {\n dim3.write([i, j, k], val3d(i, j, k));\n }\n }\n}\nems.barrier();\nstopTimer(start, dims3d[0] * dims3d[1] * dims3d[2], \" First touch \");\n\n\n//------------------------------------------------------------------------------\n// Test read & write on all nodes\nems.barrier();\nstart = new Date().getTime();\nfor (k = ems.myID; k < dims3d[2]; k += ems.nThreads) {\n for (j = 0; j < dims3d[1]; j += 1) {\n for (i = 0; i < dims3d[0]; i += 1) {\n dim3.write([i, j, k], val3d(k, j, i)); // different pattern;\n }\n }\n}\n\nfor (k = ems.myID; k < dims3d[2]; k += ems.nThreads) {\n for (j = 0; j < dims3d[1]; j += 1) {\n for (i = 0; i < dims3d[0]; i += 1) {\n assert(dim3.read([i, j, k]) === val3d(k, j, i),\n \"Failed to verify parallel 3D data \" + dim3.read([i, j, k]) +\n \" != \" + val3d(k, j, i) + \" \" + i + \" \" + j + \" \" + k);\n }\n }\n}\n\nems.barrier();\nstopTimer(start, 2 * dims3d[0] * dims3d[1] * dims3d[2], \" Read/Write ops \");\n\n\n//------------------------------------------------------------------------------\n// How long would it take node 0 alone on native array?\nems.barrier();\nstart = new Date().getTime();\nems.master(function () {\n var native = new Array(dims3d[0] * dims3d[1] * dims3d[2]);\n for (k = 0; k < dims3d[2]; k += 1) {\n for (j = 0; j < dims3d[1]; j += 1) {\n for (i = 0; i < dims3d[0]; i += 1) {\n var idx = 0;\n dims3d.forEach(function (x, i) {\n idx += x * dims3d[i]\n });\n\n native[idx] = val3d(i, j, k)\n }\n }\n }\n});\nems.barrier();\nstopTimer(start, dims3d[0] * dims3d[1] * dims3d[2], \" native array \");\n\n\n//------------------------------------------------------------------------------\n// Critical Regions\ndim1.write(30, 3333333);\nvar prev = dim1.read(30);\nems.barrier();\nstart = new Date().getTime();\n\nvar nIters = Math.floor(1000000 / ems.nThreads);\nfor (i = 0; i < nIters; i++) {\n ems.critical(function () {\n var x = dim1.read(30);\n x++;\n dim1.write(30, x);\n }, 1000); // TODO: Write proper fail case test for critical timeout\n}\n\n\nems.barrier();\nstopTimer(start, nIters * ems.nThreads, \" critical regions \");\nems.master(function () {\n assert(dim1.read(30) === (prev + (ems.nThreads * nIters)),\n \"Critical region was racing x=\" + dim1.read(30) + \" sum=\" + (prev + (ems.nThreads * nIters)) +\n \" prev=\" + prev);\n});\n\n\n//------------------------------------------------------------------------------\n// Purge D2\nstart = new Date().getTime();\nfor (j = ems.myID; j < dims2d[1]; j += ems.nThreads) {\n for (i = 0; i < dims2d[0]; i += 1) {\n dim2.writeXE([i, j], -val3d(i + 10, j + 10, 0));\n }\n}\nems.barrier();\nstopTimer(start, dims2d[0] * dims2d[1], \" writeXF purges \");\n\n\n//------------------------------------------------------------------------------\n// ReadFE then WriteEF\nstart = new Date().getTime();\n\nif (ems.myID != 0) {\n for (j = ems.myID; j < dims2d[1]; j += ems.nThreads) {\n for (i = 0; i < dims2d[0]; i += 1) {\n assert(dim2.readFF([i, j]) === val3d(i + 10, j + 10, 0),\n \"Failed to verify 2D FE data: \" +\n dim2.readFF([i, j]) + \" \" + val3d(i + 10, j + 10, 0) + \" i-j: \" + i + \" \" + j);\n }\n }\n} else {\n for (j = 0; j < dims2d[1]; j += 1) {\n for (i = 0; i < dims2d[0]; i += 1) {\n dim2.writeEF([i, j], val3d(i + 10, j + 10, 0));\n }\n }\n for (i = 0; i < dims2d[0]; i += 1) {\n assert(dim2.readFF([i, 0]) === val3d(i + 10, 10, 0),\n \"Failed to verify 2D FE node 0 data: \" +\n dim2.readFF([i, 0]) + \" \" + val3d(i + 10, 10, 0) + \" i-j: \" + i + \" \" + 0);\n }\n}\nems.barrier();\nstopTimer(start, 2 * dims2d[0] * dims2d[1], \" FE-EF Dataflow \");\n\n\n//---------------------------------------------------------------\n// Redo dataflow but using strings\nstart = new Date().getTime();\nfor (j = ems.myID; j < dims2d[1]; j += ems.nThreads) {\n for (i = 0; i < dims2d[0]; i += 1) {\n dim2.writeXE([i, j], 'mem' + (-1 * val3d(i + 10, j + 10, 0)));\n }\n}\nems.barrier();\nstopTimer(start, dims2d[0] * dims2d[1], \" XF srting purge \");\n\n\n//------------------------------------------------------------------------------\n// ReadFE then WriteEF\nstart = new Date().getTime();\n\nif (ems.myID != 0) {\n for (j = ems.myID; j < dims2d[1]; j += ems.nThreads) {\n for (i = 0; i < dims2d[0]; i += 1) {\n assert(dim2.readFF([i, j]) === 'mem' + (val3d(i + 10, j + 10, 0)),\n \"Failed to verify 2D string FE data: \" +\n dim2.readFF([i, j]) + \" \" + val3d(i + 10, j + 10, 0) + \" i-j: \" + i + \" \" + j);\n }\n }\n} else {\n for (j = 0; j < dims2d[1]; j += 1) {\n for (i = 0; i < dims2d[0]; i += 1) {\n dim2.writeEF([i, j], 'mem' + val3d(i + 10, j + 10, 0))\n }\n }\n for (i = 0; i < dims2d[0]; i += 1) {\n assert(dim2.readFF([i, 0]) === 'mem' + val3d(i + 10, 10, 0),\n \"Failed to verify 2D FE node 0 data: \" +\n dim2.readFF([i, 0]) + \" \" + val3d(i + 10, 10, 0) + \" i-j: \" + i + \" \" + 0);\n }\n}\nems.barrier();\nstopTimer(start, 2 * dims2d[0] * dims2d[1], \" Dataflow w/strgs \");\n\n\n\n"
},
{
"path": "Tests/CASdataFlow.js",
"content": "/*-----------------------------------------------------------------------------+\n | Extended Memory Semantics (EMS) Version 1.0.0 |\n | Synthetic Semantics http://www.synsem.com/ mogill@synsem.com |\n +-----------------------------------------------------------------------------+\n | Copyright (c) 2011-2014, Synthetic Semantics LLC. All rights reserved. |\n | Copyright (c) 2015-2016, Jace A Mogill. All rights reserved. |\n | |\n | Redistribution and use in source and binary forms, with or without |\n | modification, are permitted provided that the following conditions are met: |\n | * Redistributions of source code must retain the above copyright |\n | notice, this list of conditions and the following disclaimer. |\n | * Redistributions in binary form must reproduce the above copyright |\n | notice, this list of conditions and the following disclaimer in the |\n | documentation and/or other materials provided with the distribution. |\n | * Neither the name of the Synthetic Semantics nor the names of its |\n | contributors may be used to endorse or promote products derived |\n | from this software without specific prior written permission. |\n | |\n | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS |\n | \"AS IS\" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT |\n | LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR |\n | A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL SYNTHETIC |\n | SEMANTICS LLC BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, |\n | EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, |\n | PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR |\n | PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF |\n | LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING |\n | NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS |\n | SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. |\n | |\n +-----------------------------------------------------------------------------*/\n'use strict';\nvar ems = require('ems')(parseInt(process.argv[2]));\nvar assert = require('assert');\nvar start, i;\n\n\n//-------------------------------------------------------------------\n// Timer function\nfunction stopTimer(timer, nOps, label) {\n function fmtNumber(n) {\n var s = ' ' + n.toString().replace(/\\B(?=(\\d{3})+(?!\\d))/g, \",\");\n if (n < 1) {\n return n;\n }\n return s.substr(s.length - 15, s.length);\n }\n\n ems.master(function () {\n var now = new Date().getTime();\n var x = (nOps * 1000000) / ((now - timer) * 1000);\n ems.diag(fmtNumber(nOps) + label + fmtNumber(Math.floor(x).toString()) + \" ops/sec\");\n });\n}\n\n\nString.prototype.getBytes = function () {\n var bytes = [];\n for (i = 0; i < this.length; i += 1) {\n bytes.push(this.charCodeAt(i));\n }\n return bytes;\n};\n\n\n//------------------------------------------------------------------------------\n// Compare and Swap numbers\nvar casBuf = ems.new(ems.nThreads, 100000, '/tmp/EMS_3dstrings');\n\ncasBuf.writeXF(ems.myID, 100000 + ems.myID);\n\nvar v = casBuf.readFF(ems.myID);\nassert(v === 100000 + ems.myID, \"Bad first readback v=\" + v + \" computed=\" + 100000 + ems.myID);\nv = 2000000 + casBuf.readFE(ems.myID);\ncasBuf.writeEF(ems.myID, v);\nassert(casBuf.readFE(ems.myID) == 2100000 + ems.myID, \"Bad second readback\");\n\ncasBuf.writeEF(ems.myID, 33333);\ncasBuf.readFE(ems.myID);\ncasBuf.writeEF(ems.myID, 44444);\ncasBuf.readFE(ems.myID);\ncasBuf.writeEF(ems.myID, \"six\");\ncasBuf.readFF(ems.myID);\nassert(casBuf.readFF(ems.myID) === \"six\", \"Six wasn't the same... Len mem/literal\" + casBuf.readFF(ems.myID).length + \"/\" + \"six\".length );\nvar third = 'third' + ems.myID;\ncasBuf.readFE(ems.myID);\ncasBuf.writeEF(ems.myID, third);\n\nv = casBuf.readFF(ems.myID);\nassert(v === third,\n \"Bad third (string) readback v=|\" +\n v + \"| bytes:\" + v.getBytes() + \" Type=\" + typeof v + \" length=\"+ v.length + \" |||| computed=|\" +\n third + \"| bytes:\" + third.getBytes() + \" type=\" + typeof third + \" length=\"+ third.length);\n\nv = casBuf.cas(ems.myID, third, 'fourth' + ems.myID);\nassert(casBuf.readFE(ems.myID) === 'fourth' + ems.myID,\n \"Bad fourth (string) mem=\" + v + \"| old=\" + third + \"| computed=\" + 'fourth' + ems.myID);\n\ncasBuf.writeEF(ems.myID, 100 * ems.myID);\n\nassert(casBuf.readFF(ems.myID) === 100 * ems.myID, \"Failed to go from strings to numbers\");\n\nems.barrier();\n\ncasBuf.writeXF(0, 0);\nems.barrier();\n\nstart = new Date().getTime();\nvar nIters = 50;\n\nfor (i = 0; i < nIters; i += 1) {\n var oldVal = -123;\n while (oldVal != ems.myID) {\n oldVal = casBuf.cas(0, ems.myID, (ems.myID + 1) % ems.nThreads);\n }\n}\nstopTimer(start, nIters * ems.nThreads, \" CAS Numbers \");\n\n\nems.barrier();\n//------------------------------------------------------------------------------\n// Compare and Swap strings\nems.master(function () {\n casBuf.writeXF(0, 'test0');\n});\nems.barrier();\nstart = new Date().getTime();\nnIters = 5;\nvar nAMOs = 0;\n\nfor (i = 0; i < nIters; i += 1) {\n oldVal = \"no match\";\n var testMe = 'test' + ems.myID;\n var testNext = 'test' + ((ems.myID + 1) % ems.nThreads);\n while (oldVal != testMe) {\n oldVal = casBuf.cas(0, testMe, testNext);\n nAMOs++;\n }\n}\nems.barrier();\nvar rff = casBuf.readFF(0);\nassert(rff === 'test0', \"Incorrect final CAS string: \" + rff + \" test0\");\nems.diag(\"Performed \" + nAMOs + \" AMOs\");\nstopTimer(start, nIters * ems.nThreads, \" CAS Strings \");\n\nems.barrier();\n\n\n//------------------------------------------------------------------------------\n// Clobbering old casBuf definition forces destructor to be called\ncasBuf = ems.new(1, 10000, '/tmp/EMS_3dstrings'); // TODO : memory allocator for strings and objects\n"
},
{
"path": "Tests/ES6/harmony_proxies.js",
"content": "/*-----------------------------------------------------------------------------+\n | Extended Memory Semantics (EMS) Version 1.4.0 |\n | Synthetic Semantics http://www.synsem.com/ mogill@synsem.com |\n +-----------------------------------------------------------------------------+\n | Copyright (c) 2016, Jace A Mogill. All rights reserved. |\n | |\n | Redistribution and use in source and binary forms, with or without |\n | modification, are permitted provided that the following conditions are met: |\n | * Redistributions of source code must retain the above copyright |\n | notice, this list of conditions and the following disclaimer. |\n | * Redistributions in binary form must reproduce the above copyright |\n | notice, this list of conditions and the following disclaimer in the |\n | documentation and/or other materials provided with the distribution. |\n | * Neither the name of the Synthetic Semantics nor the names of its |\n | contributors may be used to endorse or promote products derived |\n | from this software without specific prior written permission. |\n | |\n | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS |\n | \"AS IS\" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT |\n | LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR |\n | A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL SYNTHETIC |\n | SEMANTICS LLC BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, |\n | EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, |\n | PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR |\n | PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF |\n | LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING |\n | NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS |\n | SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. |\n | |\n +-----------------------------------------------------------------------------*/\n\"use strict\";\nvar ems = require(\"ems\")(1);\nvar assert = require(\"assert\");\n\n// Create a temporary EMS space to demonstrate read/write operations\nvar emsData = ems.new({\n dimensions: 1000,\n heapSize: 20000,\n useMap: true,\n ES6proxies: true,\n useExisting: false\n});\n\nemsData[\"foo\"] = 123;\nassert(emsData[\"foo\"] === 123);\nassert(emsData.foo === 123);\n\nemsData.bar = 321;\nassert(emsData[\"bar\"] === 321);\nassert(emsData.bar === 321);\n\nassert(emsData.readFE(\"bar\") === 321);\nassert(emsData.bar === 321);\nemsData.writeEF(\"bar\", \"last write\");\nassert(emsData.readFE(\"bar\") === \"last write\");\nassert(emsData.bar === \"last write\");\n\nconsole.log(\"All operations passed.\");\n"
},
{
"path": "Tests/accum_omp.c",
"content": "/*-----------------------------------------------------------------------------+\n | Extended Memory Semantics (EMS) Version 1.0.0 |\n | Synthetic Semantics http://www.synsem.com/ mogill@synsem.com |\n +-----------------------------------------------------------------------------+\n | Copyright (c) 2011-2014, Synthetic Semantics LLC. All rights reserved. |\n | Copyright (c) 2015-2016, Jace A Mogill. All rights reserved. |\n | |\n | Redistribution and use in source and binary forms, with or without |\n | modification, are permitted provided that the following conditions are met: |\n | * Redistributions of source code must retain the above copyright |\n | notice, this list of conditions and the following disclaimer. |\n | * Redistributions in binary form must reproduce the above copyright |\n | notice, this list of conditions and the following disclaimer in the |\n | documentation and/or other materials provided with the distribution. |\n | * Neither the name of the Synthetic Semantics nor the names of its |\n | contributors may be used to endorse or promote products derived |\n | from this software without specific prior written permission. |\n | |\n | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS |\n | \"AS IS\" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT |\n | LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR |\n | A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL SYNTHETIC |\n | SEMANTICS LLC BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, |\n | EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, |\n | PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR |\n | PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF |\n | LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING |\n | NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS |\n | SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. |\n | |\n +-----------------------------------------------------------------------------*/\n#include \n#include \n#define SIZE 3200\n\nint main() {\n int i;\n int a[ SIZE ];\n\n for(i = 0; i < SIZE; i++) a[i] = 0;\n\n#define NITER 100000000\n\n#pragma omp parallel for\n for(i = 0; i < NITER; i++) {\n int retval = __sync_fetch_and_add( &(a[i% SIZE]), 1 );\n }\n\n\n int sum = 0;\n for(i = 0; i < SIZE; i++) {\n sum += a[i];\n }\n printf(\"sum = %d\\n\", sum);\n\n return(0);\n}\n"
},
{
"path": "Tests/barrier.js",
"content": "/*-----------------------------------------------------------------------------+\n | Extended Memory Semantics (EMS) Version 1.4.0 |\n | Synthetic Semantics http://www.synsem.com/ mogill@synsem.com |\n +-----------------------------------------------------------------------------+\n | Copyright (c) 2011-2014, Synthetic Semantics LLC. All rights reserved. |\n | Copyright (c) 2015-2016, Jace A Mogill. All rights reserved. |\n | |\n | Redistribution and use in source and binary forms, with or without |\n | modification, are permitted provided that the following conditions are met: |\n | * Redistributions of source code must retain the above copyright |\n | notice, this list of conditions and the following disclaimer. |\n | * Redistributions in binary form must reproduce the above copyright |\n | notice, this list of conditions and the following disclaimer in the |\n | documentation and/or other materials provided with the distribution. |\n | * Neither the name of the Synthetic Semantics nor the names of its |\n | contributors may be used to endorse or promote products derived |\n | from this software without specific prior written permission. |\n | |\n | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS |\n | \"AS IS\" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT |\n | LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR |\n | A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL SYNTHETIC |\n | SEMANTICS LLC BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, |\n | EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, |\n | PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR |\n | PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF |\n | LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING |\n | NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS |\n | SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. |\n | |\n +-----------------------------------------------------------------------------*/\n'use strict';\nvar ems = require('ems')(process.argv[2]);\nvar assert = require('assert');\n\nvar iter, idx, memVal;\nvar nIters = Math.floor(100000 / process.argv[2]);\nvar sums = ems.new(ems.nThreads);\nsums.writeXF(ems.myID, 0);\n\n\n\nfor (iter = 0; iter < nIters; iter += 1) {\n ems.barrier();\n idx = (ems.myID + iter) % ems.nThreads;\n memVal = sums.read(idx);\n assert(memVal === iter,\n \"myID=\" + ems.myID + \" iter=\" + iter + \" memval=\" + memVal);\n sums.write(idx, memVal + 1);\n}\nems.barrier(100);\n\n\nfor (iter = 0; iter < nIters; iter += 1) {\n ems.barrier();\n idx = (ems.myID + iter) % ems.nThreads;\n memVal = sums.read(idx);\n sums.write(idx, memVal + 1);\n}\nems.barrier();\n\nassert(sums.read(ems.myID) === (nIters * 2),\n \"myID=\" + ems.myID + \" wrong final memval=\" + sums.read(ems.myID) + \" expected=\" + (nIters * 2));\n\nems.barrier();\n\nvar shared = ems.new(process.argv[2]*2, 0, \"/tmp/EMS_mynewFoo\");\n\nshared.write(0, 0);\nems.barrier();\nvar tmp = shared.read(0);\nassert(tmp === 0, \"Didn't initialize to 0, got \" + tmp);\nems.barrier();\nshared.faa(0, 1);\nems.barrier();\nassert(shared.read(0) === ems.nThreads, \"Didn't count (\" + shared.read(0) + \") to nnodes (\" + ems.nThreads + \")\");\nems.barrier();\nif (ems.myID === 0) {\n shared.write(0, 0);\n}\nems.barrier();\nassert(shared.read(0) === 0, \"reinit Didn't initialize to 0, got \" + shared.read(0));\nems.barrier();\n\nvar nIter = 10000;\nvar i;\nfor (i = 0; i < nIter; i += 1) {\n shared.faa(0, 1);\n}\nems.barrier();\nassert(shared.read(0) === ems.nThreads * nIter, \"Random wait FAA failed: count (\" + shared.read(0) + \") to nnodes (\" + nIter + \")\");\n\nems.barrier();\nif (ems.myID === 0) {\n shared.write(0, 0);\n}\nems.barrier();\nif (ems.myID !== 0) {\n assert(shared.read(0) === 0, \"Didn't pass sync after clearing\");\n}\n\nems.barrier();\n// nIter = 10000 / process.argv[2];\n\nfor (i = 0; i < nIter; i += 1) {\n shared.faa(0, 1);\n}\nems.barrier();\nvar sr = shared.read(0);\nems.barrier();\nassert(sr === ems.nThreads * nIter, \"Fast Looped FAA failed: count (\" + sr + \") to nnodes (\" + nIter + \")\");\n\n\nvar timeout = -1;\ntry {\n timeout = ems.barrier(0);\n console.log(\"Barrier timeout succeeded but should have failed\");\n} catch (err) {\n ems.diag(\"Correctly timed out at barrier:\", err);\n}\nassert(timeout <= 0);\n"
},
{
"path": "Tests/check_image_files.sh",
"content": "#!/bin/bash\n# -*- coding: utf-8, tab-width: 2 -*-\n\n\nfunction check_image_files () {\n local SELFPATH=\"$(readlink -m \"$BASH_SOURCE\"/..)\"\n cd -- \"$SELFPATH\"/.. || return $?\n\n local IMG_DIFF=\"$(diff -sU 0 <(find_images_used) <(find_images_available\n ) | sed -re '1d;2d;/^@@/d' | norm_sort_paths)\"\n local IMG_UNUSED=() IMG_MISS=()\n\n readarray -t IMG_UNUSED < <(<<<\"$IMG_DIFF\" sed -nre 's!^\\+!!p')\n echo \"unused images: ${#IMG_UNUSED[@]}\"\n printf ' * %s\\n' \"${IMG_UNUSED[@]}\"\n\n readarray -t IMG_MISS < <(<<<\"$IMG_DIFF\" sed -nre 's!^\\-!!p')\n local N_MISS=\"${#IMG_MISS[@]}\"\n [ \"$N_MISS\" == 0 ] || exec >&2\n echo \"missing images: $N_MISS\"\n printf ' ! %s\\n' \"${IMG_MISS[@]}\"\n\n [ \"$N_MISS\" == 0 ] || return 4\n}\n\n\nfunction norm_sort_paths () {\n sed -re '\n s~(^|/)\\./~\\1~g\n ' | sort --version-sort --unique\n}\n\n\nfunction find_images_available () {\n local FIND_OPT=(\n .\n -type f\n '(' -false\n -o -name '*.png'\n -o -name '*.svg'\n -o -name '*.jpeg'\n ')'\n )\n find \"${FIND_OPT[@]}\" | norm_sort_paths\n}\n\n\nfunction find_images_used () {\n local HTML_FILES=()\n readarray -t HTML_FILES < <(find . -name '*.html')\n grep -HoPe ']*>' -- \"${HTML_FILES[@]}\" | sed -re '\n s~\\s+~ ~g\n s~= end) {\n console.log(\"Index out of bounds?!\");\n }\n for (var i = 0; i < nIters; i++) {\n sum++;\n }\n}\n\nstats.writeXF(0, 0);\nems.barrier();\nstartTime = util.timerStart();\nems.parForEach(start, end, doWork);\nstats.faa(0, sum);\nutil.timerStop(startTime, n * ems.nThreads, \" Default \", ems.myID);\nems.barrier();\nems.diag(\"The sum: \" + sum);\nems.barrier();\nsum = stats.read(0);\nassert(sum == n, \"Default scheduling failed sum=\" + sum + \" expected \" + n);\n\n\nsum = 0;\nstartTime = util.timerStart();\nems.parForEach(start, end, doWork, 'dynamic');\nutil.timerStop(startTime, n * ems.nThreads, \" Dynamic \", ems.myID);\nems.barrier();\nems.diag(\"The sum: \" + sum);\nems.barrier();\nsum = stats.read(0);\nassert(sum == n, \"Dynamic scheduling failed sum=\" + sum + \" expected \" + n);\n\n\nsum = 0;\nstartTime = util.timerStart();\nems.parForEach(start, end, doWork, 'static');\nutil.timerStop(startTime, n * ems.nThreads, \" Static \", ems.myID);\nems.barrier();\nems.diag(\"The sum: \" + sum);\nems.barrier();\nsum = stats.read(0);\nassert(sum == n, \"Static scheduling failed sum=\" + sum + \" expected \" + n);\n\n\nsum = 0;\nstartTime = util.timerStart();\nems.parForEach(start, end, doWork, 'guided', 20);\nutil.timerStop(startTime, n * ems.nThreads, \" Guided by 20 \", ems.myID);\nems.barrier();\nems.diag(\"The sum: \" + sum);\nems.barrier();\nsum = stats.read(0);\nassert(sum == n, \"Guided scheduling failed sum=\" + sum + \" expected \" + n);\n"
},
{
"path": "Tests/mapped_test.js",
"content": "/*-----------------------------------------------------------------------------+\n | Extended Memory Semantics (EMS) Version 1.3.0 |\n | Synthetic Semantics http://www.synsem.com/ mogill@synsem.com |\n +-----------------------------------------------------------------------------+\n | Copyright (c) 2011-2014, Synthetic Semantics LLC. All rights reserved. |\n | Copyright (c) 2015-2016, Jace A Mogill. All rights reserved. |\n | |\n | Redistribution and use in source and binary forms, with or without |\n | modification, are permitted provided that the following conditions are met: |\n | * Redistributions of source code must retain the above copyright |\n | notice, this list of conditions and the following disclaimer. |\n | * Redistributions in binary form must reproduce the above copyright |\n | notice, this list of conditions and the following disclaimer in the |\n | documentation and/or other materials provided with the distribution. |\n | * Neither the name of the Synthetic Semantics nor the names of its |\n | contributors may be used to endorse or promote products derived |\n | from this software without specific prior written permission. |\n | |\n | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS |\n | \"AS IS\" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT |\n | LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR |\n | A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL SYNTHETIC |\n | SEMANTICS LLC BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, |\n | EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, |\n | PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR |\n | PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF |\n | LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING |\n | NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS |\n | SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. |\n | |\n +-----------------------------------------------------------------------------*/\n'use strict';\nvar ems = require('ems')(parseInt(process.argv[2]), false);\nvar assert = require('assert');\nvar arrayElem = ['abcd', 1234.567, true, 987, -1234.567, -987, -986, -988];\nvar arrLen = arrayElem.length;\n\nvar objMap = ems.new({\n dimensions: [arrLen],\n heapSize: arrLen * 200,\n useMap: true,\n useExisting: false,\n setFEtags: 'full'\n});\n\n\narrayElem.forEach(function (elem, idx) {\n if (ems.myID === 0) { console.log(\"Trying to map:\", elem, typeof elem); }\n objMap.writeXF(elem, elem + idx);\n var readback = objMap.readFF(elem);\n assert(readback === elem + idx, 'Readback of ' + (elem + idx) + 'didnt match:' + readback);\n});\n"
},
{
"path": "Tests/py_api.py",
"content": "# -*- coding: utf-8 -*-\n\"\"\"\n +-----------------------------------------------------------------------------+\n | Extended Memory Semantics (EMS) Version 1.4.1 |\n | Synthetic Semantics http://www.synsem.com/ mogill@synsem.com |\n +-----------------------------------------------------------------------------+\n | Copyright (c) 2016, Jace A Mogill. All rights reserved. |\n | |\n | Redistribution and use in source and binary forms, with or without |\n | modification, are permitted provided that the following conditions are met: |\n | * Redistributions of source code must retain the above copyright |\n | notice, this list of conditions and the following disclaimer. |\n | * Redistributions in binary form must reproduce the above copyright |\n | notice, this list of conditions and the following disclaimer in the |\n | documentation and/or other materials provided with the distribution. |\n | * Neither the name of the Synthetic Semantics nor the names of its |\n | contributors may be used to endorse or promote products derived |\n | from this software without specific prior written permission. |\n | |\n | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS |\n | \"AS IS\" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT |\n | LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR |\n | A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL SYNTHETIC |\n | SEMANTICS LLC BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, |\n | EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, |\n | PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR |\n | PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF |\n | LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING |\n | NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS |\n | SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. |\n | |\n +-----------------------------------------------------------------------------+\n\"\"\"\nimport sys\nimport os\nimport time\nsys.path.append(\"../Python/ems/\") # Path to EMS Python Module\nimport ems\n\nnprocs = 12\nnelem = 1000\nems.initialize(nprocs, False, 'bsp', '/tmp/pyems_foo_test.ems')\ntime.sleep(ems.myID/5)\nassert ems.barrier()\n\n\nvarious_consts = [\"簣ひょ갤妣ゃキェぱ覣.𐤦ぴ盥\", -99999, 1 << 35, 9821.3456, \"bar\", False]\nunmapped_fname = '/tmp/py_unmapped.ems'\n# TODO: 3D array\nunmapped = ems.new(nelem * nprocs, nprocs * nelem * 100, unmapped_fname)\nems.diag(\"=============== INIT COMPLETE ======================\")\n\n\ndef sum(left, right):\n if type(left) == int:\n return right + left\n if type(left) == float:\n return right * left\n else:\n return str(right) + str(left)\n\n\ndef target(idx):\n return idx + (ems.myID * nelem)\n\n\nfor const in various_consts:\n for idx in range(nelem):\n unmapped.writeXF(target(idx), sum(const, idx))\n\n for idx in range(nelem):\n val = unmapped.readFE(target(idx))\n if sum(const, idx) != val:\n ems.diag(\"I'm lost... sum=%s val=%s idx=%d target=%d\" % (sum(const, idx), val, idx, target(idx)))\n assert sum(const, idx) == val\nems.barrier()\nems.diag(\"Starting mapped tests\")\narrLen = 1000\nmapped_fname = '/tmp/py_mapped.ems'\nmapped = ems.new({\n 'dimensions': [arrLen],\n 'heapSize': arrLen * 200,\n 'useMap': True,\n 'useExisting': False,\n 'filename': mapped_fname,\n 'doSetFEtags': True,\n 'setFEtags': 'empty',\n 'dataFill': None\n})\n\narrayElem = ['abcd', True, 1234.567, False, {'x': 'xxx', 'y': 'yyyyy'}, 987, None, [10, 11, 12, 13]]\nobjElem = {'a': 1, 'b': 321.653, 'x': {'nesta': False, 'nestb': None}, 'c': 'asdasd'}\n\nteststr = 'magic!!string?!'\n\nmapped.writeEF(teststr, ems.myID)\nassert ems.myID == mapped.readFE(teststr)\n# ems.diag(\"First mapped write passed\")\n\nmapped.writeEF(teststr + \"XXX\", ems.myID * 100)\nassert ems.myID * 100 == mapped.readFE(teststr + \"XXX\")\n# ems.diag(\"Second mapped write passed\")\n\nmapped.writeEF(ems.myID, teststr)\n# ems.diag(\"Wrote the test string\")\nassert teststr == mapped.readFE(ems.myID)\n\n# Check read/writing JSON\nmapped.writeEF(ems.myID, objElem)\ntmp = mapped.readFE(ems.myID)\n# ems.diag(\"ObjElem READback\" + str(tmp))\nassert objElem == tmp\n\nmapped.writeEF(ems.myID, arrayElem)\ntmp = mapped.readFE(ems.myID)\n# ems.diag(\"READback:\" + str(mapped.readFE(ems.myID)))\nassert arrayElem == tmp\ntmp = mapped.read(ems.myID)\nassert arrayElem == tmp\n\nmapped.write(ems.myID, 'dummy')\nassert 'dummy' == mapped.read(ems.myID)\n\n# ==========================================================================\ndef parfun(idx):\n i = mapped.faa('index sum', idx - start + 1)\n c = mapped.faa('counter', 1)\n timeout = 10000\n while (timeout > 0) and mapped.cas('cas test', None, 'something') is not None:\n timeout -= 1\n assert timeout > 0\n assert 'something' == mapped.cas('cas test', 'something', None)\n # ems.diag(\"PARFUN: idx(%d) i(%d) c(%d)\" % (idx, i, c))\n\n\ndef triangular(max = 10):\n for top in range(1, max):\n x = 0\n for idx in range(1, top+1):\n x += idx\n print(\"top=\", top, \" x=\", x, \" calc=\", (top * (top + 1)) / 2)\n assert x == (top * (top + 1)) / 2\n\n\ndef check_tri(msg):\n index_sum = mapped.readFF('index sum')\n count = mapped.readFF('counter')\n # ems.diag(msg + \": sum=%d expected=%d\" % (index_sum, (niters * (niters + 1))/2))\n assert index_sum == trisum\n assert count == niters\n reset_tri()\n\n\ndef reset_tri():\n ems.barrier()\n mapped.writeXF('index sum', 0)\n mapped.writeXF('counter', 0)\n mapped.writeXF('cas test', None)\n ems.barrier()\n\nunmapped.writeXF(ems.myID, 1000+ems.myID)\nems.single(triangular)\nidx = (ems.myID+1) % nprocs\nassert unmapped.readFF(idx) == 1000+idx\n\n# Scheduling needs proper testing other than visual inspection\nstart = 100\nend = 120\nniters = end - start\ntrisum = (niters * (niters + 1))/2\n\nreset_tri()\nems.parForEach(start, end, parfun)\ncheck_tri(\"default\")\nems.parForEach(start, end, parfun, 'static')\ncheck_tri(\"static\")\nems.parForEach(start, end, parfun, 'guided')\ncheck_tri(\"guided\")\nems.parForEach(start, end, parfun, 'dynamic')\ncheck_tri(\"dynamic\")\n\n# ==========================================================================\nunmapped.writeXF(0, True)\nunmapped.writeXF(1, 0)\nmapped.writeXF('one', 0)\nems.barrier()\ntm_data = [[unmapped, 1], [mapped, 'one'], [unmapped, 0, True]]\nfor idx in range(niters):\n transaction = ems.tmStart(tm_data)\n unmapped_tmp = unmapped.read(1)\n mapped_tmp = mapped.read('one')\n assert unmapped.read(0)\n unmapped.write(1, unmapped_tmp + 1)\n mapped.write('one', mapped_tmp + 1)\n ems.tmEnd(transaction, True)\n\nems.barrier()\nassert unmapped.readFF(1) == nprocs * niters\nassert mapped.readFF('one') == nprocs * niters\nems.barrier()\n\n\n# ==========================================================================\n\n\ndef plus_one():\n tmp = unmapped.read(0)\n unmapped.write(0, tmp + 1)\n\nems.barrier()\nunmapped.writeXF(0, 0)\nems.barrier()\nems.critical(plus_one)\nreadback = unmapped.readFF(0)\n# ems.diag(\"Initial Readback before critical: %d\" % readback)\nif readback == nprocs:\n unmapped.writeXF(0, -1234)\nems.barrier()\nreadback = unmapped.readFF(0)\n# ems.diag(\"Readback of critical: %d\" % readback)\nassert readback == -1234\n\n\n# ==========================================================================\ndef check_master():\n assert ems.myID == 0\n\nems.master(check_master)\n\n\n# ==========================================================================\ndef check_single():\n global idx\n oldval = mapped.cas(idx, None, 'Wheee!!!')\n if oldval is None:\n assert mapped.faa('count', 1) == 0\n\nidx = 987.654\nems.barrier()\nmapped.writeXF(idx, None)\nmapped.writeXF('count', 0)\nems.barrier()\nems.single(check_single)\nassert mapped.readFF(idx) == 'Wheee!!!'\nassert mapped.readFF('count') == 1\n\n\n# ==========================================================================\nkeys = [mapped.index2key(idx) for idx in range(arrLen)]\n# print(\"KEYS!\", keys)\nassert 'count' in keys\nassert 'counter' in keys\nassert teststr in keys\nassert teststr+'XXX' in keys\n\n\n# ==========================================================================\nstack = ems.new({\n 'dimensions': [arrLen],\n 'heapSize': arrLen * 100,\n 'useExisting': False,\n 'filename': '/tmp/stack_test.ems',\n 'doSetFEtags': True,\n 'setFEtags': 'empty',\n 'dataFill': None\n})\n\nfor elem in arrayElem:\n assert stack.push(elem) >= 0\n\nfor elem in arrayElem:\n assert stack.pop() in arrayElem\n\nems.barrier()\nassert stack.pop() is None\nems.barrier()\n\n\nfor elem in arrayElem:\n assert stack.enqueue(elem) >= 0\n\nfor elem in arrayElem:\n assert stack.dequeue() in arrayElem\n\nems.barrier()\nassert stack.dequeue() is None\nems.barrier()\n\n# ==========================================================================\n# Fancy array syntax\nmapped.writeXF(-1234, 'zero')\nmapped[-1234].writeXF(0)\nidx = 'fancy' + str(ems.myID)\nval = 123 + ems.myID\nrw_val = 12345 + ems.myID\nrw_idx = 'rw test' + str(ems.myID)\nmapped[rw_idx].writeXF(rw_val)\nmapped[idx].writeXF(val)\n# print(\"Fancyread\", ems.myID, mapped[idx].read(), mapped[idx].xread)\nassert val == mapped[idx].read()\nassert val == mapped[idx].readFF()\nassert val == mapped[idx].readFE()\nval *= 200\nmapped[idx].writeEF(val)\nassert val == mapped[idx].readFF()\nems.barrier()\nassert rw_val == mapped[rw_idx].readRW()\nassert mapped[rw_idx].releaseRW() >= 0\nems.barrier()\nval *= 200\nmapped[idx].write(val)\nassert val == mapped[idx].read()\nassert 0 <= mapped[-1234].faa(1) < nprocs\nif ems.myID == 0:\n mapped[-1234].cas(nprocs-1, True)\nassert mapped[-1234].read()\n\nmapped[idx].writeXE(123)\nmapped[idx].setTag('full')\n\nif ems.myID == 0:\n mapped.writeXF('syntax_test', 123)\n assert mapped.readFE('syntax_test') == 123\n assert mapped['syntax_test'].read() == 123\n assert mapped.syntax_test == 123\n\n mapped['syntax_test2'] = 234\n mapped['syntax_test2'].setTag('full')\n # assert mapped.readFE('syntax_test2') == 234\n mapped['syntax_test3'].writeXF(234)\n assert mapped.readFE('syntax_test2') == 234\n\nems.barrier()\n\n\n'''\nassert mapped.fancy0 == 123\nassert unmapped[0] == -123\n\nems.barrier()\nif ems.myID == 0:\n mapped.fizz = 'asd'\nems.barrier()\nassert mapped.fizz == 'asd'\n'''\n\n\n# ==========================================================================\n\nunmapped.destroy(True)\n# ems.diag(\"UNMAPPING:\" + unmapped_fname + \" Exist?\" + os.path.exists(unmapped_fname))\nassert not os.path.exists(unmapped_fname)\nmapped.destroy(False)\nassert os.path.exists(mapped_fname)\n\nmapped = ems.new({\n 'dimensions': [arrLen],\n 'heapSize': arrLen * 200,\n 'useMap': True,\n 'useExisting': True,\n 'filename': mapped_fname,\n 'doSetFEtags': False,\n 'doDataFill': False\n})\nassert mapped.readFF(idx) == 123\n"
},
{
"path": "Tests/readers-writer.js",
"content": "/*-----------------------------------------------------------------------------+\n | Extended Memory Semantics (EMS) Version 1.3.0 |\n | Synthetic Semantics http://www.synsem.com/ mogill@synsem.com |\n +-----------------------------------------------------------------------------+\n | Copyright (c) 2011-2014, Synthetic Semantics LLC. All rights reserved. |\n | Copyright (c) 2015-2016, Jace A Mogill. All rights reserved. |\n | |\n | Redistribution and use in source and binary forms, with or without |\n | modification, are permitted provided that the following conditions are met: |\n | * Redistributions of source code must retain the above copyright |\n | notice, this list of conditions and the following disclaimer. |\n | * Redistributions in binary form must reproduce the above copyright |\n | notice, this list of conditions and the following disclaimer in the |\n | documentation and/or other materials provided with the distribution. |\n | * Neither the name of the Synthetic Semantics nor the names of its |\n | contributors may be used to endorse or promote products derived |\n | from this software without specific prior written permission. |\n | |\n | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS |\n | \"AS IS\" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT |\n | LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR |\n | A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL SYNTHETIC |\n | SEMANTICS LLC BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, |\n | EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, |\n | PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR |\n | PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF |\n | LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING |\n | NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS |\n | SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. |\n | |\n +-----------------------------------------------------------------------------*/\nvar assert = require('assert');\nvar ems = require('ems')(parseInt(process.argv[2]), false);\nvar a = ems.new(1, 0, '/tmp/EMS_a');\nvar arrLen = 10000;\nvar nTimes;\nvar nLocks;\nvar count = ems.new({\n dimensions: [10],\n heapSize: arrLen * 200,\n useMap: true,\n useExisting: false,\n setFEtags: 'full'\n});\n\na.writeXF(0, 123);\ncount.writeXF(0, 0);\nems.barrier();\n\nfor (nTimes = 0; nTimes < 100000; nTimes += 1) {\n a.readRW(0);\n nLocks = count.faa(0, 1) + 1;\n assert(nLocks <= 7, \"Too many locks: \" + nLocks);\n nLocks = count.faa(0, -1) - 1;\n assert(nLocks >= 0, \"Too few locks: \" + nLocks);\n for (var i = 0; i < 100; i++) {\n nLocks += Math.sin(i);\n }\n a.releaseRW(0);\n}\n\nems.barrier();\n\nvar objMap = ems.new({\n dimensions: [arrLen],\n heapSize: arrLen * 200,\n useMap: true,\n useExisting: false,\n setFEtags: 'full'\n});\n\n['abcd', 1234.567, true, 987].forEach( function(elem) {\n objMap.writeXF(elem, elem);\n count.writeXF(elem, 0);\n});\nems.barrier();\n\n['abcd', 1234.567, true, 987].forEach( function(elem) {\n for (var nTimes = 0; nTimes < 10000; nTimes++) {\n var readback = objMap.readRW(elem);\n assert(readback === elem,\n \"Multi Reader read wrong data. Expected(\" + elem + \"), got(\" + readback + \")\");\n var nLocks = count.faa(elem, 1) + 1;\n assert(nLocks <= 7, \"Too many locks: \" + nLocks);\n nLocks = count.faa(elem, -1) - 1;\n assert(nLocks >= 0, \"Too few locks: \" + nLocks);\n for (var i = 0; i < nLocks * 100; i += 1) {\n nLocks += Math.sin(i);\n }\n objMap.releaseRW(elem);\n }\n});\n"
},
{
"path": "Tests/refactoring.js",
"content": "/*-----------------------------------------------------------------------------+\n | Extended Memory Semantics (EMS) Version 1.4.0 |\n | Synthetic Semantics http://www.synsem.com/ mogill@synsem.com |\n +-----------------------------------------------------------------------------+\n | Copyright (c) 2011-2014, Synthetic Semantics LLC. All rights reserved. |\n | Copyright (c) 2015-2016, Jace A Mogill. All rights reserved. |\n | |\n | Redistribution and use in source and binary forms, with or without |\n | modification, are permitted provided that the following conditions are met: |\n | * Redistributions of source code must retain the above copyright |\n | notice, this list of conditions and the following disclaimer. |\n | * Redistributions in binary form must reproduce the above copyright |\n | notice, this list of conditions and the following disclaimer in the |\n | documentation and/or other materials provided with the distribution. |\n | * Neither the name of the Synthetic Semantics nor the names of its |\n | contributors may be used to endorse or promote products derived |\n | from this software without specific prior written permission. |\n | |\n | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS |\n | \"AS IS\" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT |\n | LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR |\n | A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL SYNTHETIC |\n | SEMANTICS LLC BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, |\n | EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, |\n | PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR |\n | PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF |\n | LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING |\n | NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS |\n | SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. |\n | |\n +-----------------------------------------------------------------------------*/\n'use strict';\nvar util = require('util');\nvar assert = require('assert');\nvar ems = require('ems')(1, 0, 'bsp', '/facto.ems');\nvar array_len = 10000;\nvar ems_config = {\n dimensions : [array_len],\n heapSize : 20000,\n useMap : false,\n useExisting : false,\n filename : '/tmp/refactor.ems'\n};\n\nvar sums = ems.new(ems_config);\n\nems_config.useMap = true;\nems_config.filename = '/tmp/anotherplace';\nvar emptyland = ems.new(ems_config);\n\nems.barrier();\n/*\nsetTimeout(function () {\n // console.log(\"Pre timeout barrier\", ems.myID);\n ems.barrier();\n console.log(\"Post timeout barrier\", ems.myID);\n}, ems.myID * 3000);\n*/\n\nvar magicval = 0x15432100 + ems.myID;\nsums.writeXF(ems.myID, magicval);\nvar readback = sums.readFF(ems.myID);\nassert(readback === magicval, \"Expected \" + magicval.toString() + \", got \" + readback.toString());\nsums.readFF(ems.myID);\n\nems.barrier();\nems.barrier();\n\nsums.writeXE(ems.myID, \"Nothing to see\");\nvar idx, teststr;\nfor (idx = 0; idx < 100000; idx += 1) {\n teststr = \"some text\" + idx;\n sums.writeEF(ems.myID, teststr);\n readback = sums.readFE(ems.myID);\n assert(teststr === readback, \"Nope...|\" + teststr + \"|\" + readback + \"| len=\" + teststr.length + \"/\" + readback.length);\n}\n\nfor (idx = 0; idx < 100000; idx += 1) {\n teststr = 'foo' + (idx * 3 + idx * 7);\n readback = emptyland.read(teststr);\n assert(readback === undefined, \"Read actual data during undefined map test:\" + teststr);\n}\n\nvar oldval;\nfor (idx = 0; idx < array_len / 40; idx += 1) {\n teststr = 'foo' + (idx * 5) + (idx * array_len);\n\n readback = emptyland.read(teststr);\n assert(readback === undefined, \"CAS phase: Read actual data during undefined map test:\" + teststr);\n readback = emptyland.read(teststr);\n assert(readback === undefined, \"CAS phase: Second Read actual data during undefined map test:\" + teststr);\n\n oldval = emptyland.cas(teststr, undefined, teststr);\n assert(oldval === undefined, \"CAS at teststr(\" + teststr + \") wasn't undefined -- value was:\" + oldval + \"/\" + (oldval || 0xf00dd00d).toString(16));\n var newmemval = emptyland.readFF(teststr);\n assert(teststr === newmemval, \"wrong post CAS mem value -- value was:\" + newmemval + \" expected:\" + teststr);\n}\n\n\n// Check index2key\nvar all_types_len = 10;\nvar all_types = ems.new({\n dimensions : [ all_types_len ],\n heapSize : 10000, // Optional, default=0: Space, in bytes, for\n useMap : true, // Optional, default=false: Map keys to indexes\n filename : '/tmp/refactor.ems' // Optional, default=anonymous:\n});\n\nvar all_data_types = [false, true, 1234, 987.654321, 'hello'];\nall_data_types.forEach(function(type) {\n console.log(\"Trying type\", type);\n all_types.writeXF(type, type);\n});\n\nfor(idx = 0; idx < all_types_len; idx += 1) {\n console.log(\"Index:\" + idx + \" -- key:\" + all_types.index2key(idx));\n}\n"
},
{
"path": "Tests/stack_and_queue.js",
"content": "/*-----------------------------------------------------------------------------+\n | Extended Memory Semantics (EMS) Version 1.4.0 |\n | Synthetic Semantics http://www.synsem.com/ mogill@synsem.com |\n +-----------------------------------------------------------------------------+\n | Copyright (c) 2011-2014, Synthetic Semantics LLC. All rights reserved. |\n | Copyright (c) 2015-2016, Jace A Mogill. All rights reserved. |\n | |\n | Redistribution and use in source and binary forms, with or without |\n | modification, are permitted provided that the following conditions are met: |\n | * Redistributions of source code must retain the above copyright |\n | notice, this list of conditions and the following disclaimer. |\n | * Redistributions in binary form must reproduce the above copyright |\n | notice, this list of conditions and the following disclaimer in the |\n | documentation and/or other materials provided with the distribution. |\n | * Neither the name of the Synthetic Semantics nor the names of its |\n | contributors may be used to endorse or promote products derived |\n | from this software without specific prior written permission. |\n | |\n | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS |\n | \"AS IS\" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT |\n | LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR |\n | A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL SYNTHETIC |\n | SEMANTICS LLC BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, |\n | EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, |\n | PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR |\n | PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF |\n | LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING |\n | NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS |\n | SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. |\n | |\n +-----------------------------------------------------------------------------*/\n'use strict';\nvar ems = require('ems')(parseInt(process.argv[2]));\nvar assert = require('assert');\nvar arrLen = 1000000;\nvar a = ems.new({\n dimensions: [arrLen],\n heapSize: arrLen * 100,\n useExisting: false,\n filename: '/tmp/EMS_stack',\n doSetFEtags: true,\n setFEtags: 'empty',\n doSetDataFill: true,\n dataFill: undefined\n});\n\nvar b = ems.new({\n dimensions: [arrLen],\n heapSize: arrLen * 100,\n doSetFEtags: true,\n setFEtags: 'empty',\n doSetDataFill: true,\n dataFill: undefined\n});\n\nvar timeStart, tmp, i, idx;\n\n\n//-------------------------------------------------------------------\n// Timer function\nfunction stopTimer(timer, nOps, label) {\n function fmtNumber(n) {\n var s = ' ' + n.toString().replace(/\\B(?=(\\d{3})+(?!\\d))/g, \",\");\n if (n < 1) return n;\n else {\n return s.substr(s.length - 15, s.length);\n }\n }\n\n ems.master(function () {\n var now = new Date().getTime();\n var x = (nOps * 1000000) / ((now - timer) * 1000);\n ems.diag(fmtNumber(nOps) + label + fmtNumber(Math.floor(x).toString()) + \" ops/sec\");\n });\n}\n\nvar x = a.pop();\nassert(x === undefined, \"Initial: should have been empty: \" + x);\nems.barrier();\n\na.push(1000 + ems.myID);\nems.barrier();\nx = a.pop();\nassert(x >= 1000, \"Expected something >= 1000 on the stack, found \" + x);\nems.barrier();\nx = a.pop();\nassert(x === undefined, \"Expected nothing on the stack, found: \" + x);\n\nems.barrier();\na.push(2000.12345 + ems.myID);\na.push('text ' + ems.myID + 'yeah...');\nems.barrier();\nassert(a.pop() !== undefined, \"Expected text or ### on the stack, was undefined.\")\nassert(a.pop() !== undefined, \"Expected second text or ### on the stack, was undefined.\")\nems.barrier();\nassert(a.pop() === undefined, \"after two pops stack should have been empty\");\nems.barrier();\na.push(3330.12345 + ems.myID);\na.push('more text ' + ems.myID + 'yeah...');\nassert(a.pop() !== undefined, \"Expected more text or more ### on the stack, was undefined.\")\nassert(a.pop() !== undefined, \"Expected second more text or more ### on the stack, was undefined.\")\nems.barrier();\nassert(a.pop() === undefined, \"repeated two pops should have been empty\");\n\nvar value = b.dequeue();\nems.barrier();\nassert(value === undefined, \"Intitial b dequeue should have beeen undefined, was \" + value);\nems.barrier();\nidx = b.enqueue(4000 + ems.myID);\n\nems.barrier();\nems.master(function () {\n tmp = b.dequeue();\n var count = 0;\n while (tmp !== undefined) {\n tmp = b.dequeue();\n count++;\n }\n assert(count === ems.nThreads, \"Didn't find enough queued items\");\n assert(a.pop() === undefined, \"dq1: should still have been empty\");\n});\nems.barrier();\ntmp = b.dequeue();\nassert(tmp === undefined, \"DQ should be mt: \" + tmp);\nems.barrier();\nems.master(function () {\n for (i = 0; i < arrLen; i++) {\n var str = \"sequential\" + (i * 100);\n b.enqueue(str);\n }\n for (i = 0; i < arrLen; i++) {\n tmp = b.dequeue();\n assert(tmp === \"sequential\" + (i * 100), \"Iter \" + i + \" Should have found ... got \" + tmp);\n }\n assert(a.pop() === undefined, \"a pop again should have been empty\");\n});\nems.barrier();\n\ntimeStart = new Date().getTime();\nems.parForEach(0, arrLen, function (idx) {\n b.enqueue('foo' + idx);\n});\nstopTimer(timeStart, arrLen, \" enqueued \");\n\ntimeStart = new Date().getTime();\nems.parForEach(0, arrLen, function () {\n b.dequeue();\n});\nstopTimer(timeStart, arrLen, \" dequeued \");\n\n\ntmp = b.dequeue();\nassert(tmp === undefined, \"DQ at end should be mt: \" + tmp);\n\nvar p = ems.new({\n dimensions: [arrLen + 1],\n heapSize: arrLen * 50,\n doSetFEtags: true,\n setFEtags: 'empty',\n dataFill: undefined\n});\ntimeStart = new Date().getTime();\nems.parForEach(0, arrLen, function (idx) {\n p.push('foo' + idx);\n});\nstopTimer(timeStart, arrLen, \" pushed \");\n\ntimeStart = new Date().getTime();\nems.parForEach(0, arrLen, function () {\n p.pop();\n});\nstopTimer(timeStart, arrLen, \" popped \");\n\nems.barrier();\nvar c = [];\ntimeStart = new Date().getTime();\nems.master(function () {\n for (idx = 0; idx < arrLen; idx++) {\n c.push('foo' + idx);\n }\n});\nstopTimer(timeStart, arrLen, \" native enqueued \");\n\ntimeStart = new Date().getTime();\nems.master(function () {\n for (idx = 0; idx < arrLen; idx++) {\n c.pop();\n }\n});\nstopTimer(timeStart, arrLen, \" native dequeued \");\n\ntmp = b.dequeue();\nassert(tmp === undefined, \"DQ of B after native end should be mt: \" + tmp);\n\n/////////////////////////////////////////\n\ntimeStart = new Date().getTime();\nems.parForEach(0, arrLen, function (idx) {\n b.enqueue(idx);\n});\nstopTimer(timeStart, arrLen, \" int enqueued \");\n\ntimeStart = new Date().getTime();\nems.parForEach(0, arrLen, function () {\n b.dequeue();\n});\nstopTimer(timeStart, arrLen, \" int dequeued \");\n\n\ntmp = b.dequeue();\nassert(tmp === undefined, \"DQend second time should be mt: \" + tmp);\n\n\np = ems.new({\n dimensions: [arrLen + 1],\n heapSize: arrLen * 50,\n doSetFEtags: true,\n setFEtags: 'empty',\n dataFill: undefined\n});\ntimeStart = new Date().getTime();\nems.parForEach(0, arrLen, function (idx) {\n p.push(idx);\n});\nstopTimer(timeStart, arrLen, \" int pushed \");\n\ntimeStart = new Date().getTime();\nems.parForEach(0, arrLen, function () {\n p.pop();\n});\nstopTimer(timeStart, arrLen, \" int popped \");\n\nems.barrier();\nc = [];\ntimeStart = new Date().getTime();\nems.master(function () {\n for (idx = 0; idx < arrLen; idx++) {\n c.push(idx);\n }\n});\nstopTimer(timeStart, arrLen, \" int native enqueued \");\n\ntimeStart = new Date().getTime();\nems.master(function () {\n for (idx = 0; idx < arrLen; idx++) {\n c.pop();\n }\n});\nstopTimer(timeStart, arrLen, \" int native dequeued \");\n\ntmp = b.dequeue();\nassert(tmp === undefined, \"Final deque should be mt: \" + tmp);\n\n"
},
{
"path": "Tests/strcpy.js",
"content": "/*-----------------------------------------------------------------------------+\n | Extended Memory Semantics (EMS) Version 1.5.0 |\n | Synthetic Semantics http://www.synsem.com/ mogill@synsem.com |\n +-----------------------------------------------------------------------------+\n | Copyright (c) 2011-2014, Synthetic Semantics LLC. All rights reserved. |\n | Copyright (c) 2015-2017, Jace A Mogill. All rights reserved. |\n | |\n | Redistribution and use in source and binary forms, with or without |\n | modification, are permitted provided that the following conditions are met: |\n | * Redistributions of source code must retain the above copyright |\n | notice, this list of conditions and the following disclaimer. |\n | * Redistributions in binary form must reproduce the above copyright |\n | notice, this list of conditions and the following disclaimer in the |\n | documentation and/or other materials provided with the distribution. |\n | * Neither the name of the Synthetic Semantics nor the names of its |\n | contributors may be used to endorse or promote products derived |\n | from this software without specific prior written permission. |\n | |\n | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS |\n | \"AS IS\" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT |\n | LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR |\n | A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL SYNTHETIC |\n | SEMANTICS LLC BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, |\n | EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, |\n | PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR |\n | PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF |\n | LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING |\n | NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS |\n | SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. |\n | |\n +-----------------------------------------------------------------------------*/\n'use strict';\nvar ems = require('ems')(parseInt(process.argv[2]));\nvar assert = require('assert');\nvar maxlen = 20000000;\nvar stats = ems.new({\n filename: '/tmp/EMS_strlen',\n dimensions: [10],\n heapSize: (2 * maxlen) + 1 + 10, // 1 = Null, 10 = length of key\n useMap: true, // Use a key-index mapping, not integer indexes\n setFEtags: 'full', // Initial full/empty state of array elements\n doDataFill: true, // Initialize data values\n dataFill: 0 // Initial value of new keys\n});\n\nif (ems.myID === 0) { stats.writeXE('test_str', 123); }\nems.barrier();\n\nfunction stringFill(x, n) {\n var s = '';\n for (;;) {\n if (n & 1) s += x;\n n >>= 1;\n if (n) x += x;\n else break;\n }\n return s;\n}\n\nfor (var len=2; len < maxlen; len = Math.floor(len * 1.5) ) {\n if (ems.myID === 0) { console.log(\"Len = \" + len); }\n var str = stringFill('x', len);\n stats.writeEF('test_str', str);\n var readback = stats.readFE('test_str');\n assert(readback === str, 'Mismatched string. Expected len ' + str.length + ' got ' + readback.length);\n ems.barrier();\n}\n\n\nfor (var len=maxlen; len >= 1; len = Math.floor(len * 0.666) ) {\n if (ems.myID === 0) { console.log(\"Len = \" + len); }\n var str = stringFill('y', len);\n stats.writeEF('test_str', str);\n var readback = stats.readFE('test_str');\n assert(readback === str, 'Mismatched string. Expected len ' + str.length + ' got ' + readback.length);\n ems.barrier();\n}\n"
},
{
"path": "Tests/stringTags.js",
"content": "/*-----------------------------------------------------------------------------+\n | Extended Memory Semantics (EMS) Version 1.0.0 |\n | Synthetic Semantics http://www.synsem.com/ mogill@synsem.com |\n +-----------------------------------------------------------------------------+\n | Copyright (c) 2011-2014, Synthetic Semantics LLC. All rights reserved. |\n | Copyright (c) 2015-2016, Jace A Mogill. All rights reserved. |\n | |\n | Redistribution and use in source and binary forms, with or without |\n | modification, are permitted provided that the following conditions are met: |\n | * Redistributions of source code must retain the above copyright |\n | notice, this list of conditions and the following disclaimer. |\n | * Redistributions in binary form must reproduce the above copyright |\n | notice, this list of conditions and the following disclaimer in the |\n | documentation and/or other materials provided with the distribution. |\n | * Neither the name of the Synthetic Semantics nor the names of its |\n | contributors may be used to endorse or promote products derived |\n | from this software without specific prior written permission. |\n | |\n | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS |\n | \"AS IS\" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT |\n | LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR |\n | A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL SYNTHETIC |\n | SEMANTICS LLC BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, |\n | EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, |\n | PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR |\n | PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF |\n | LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING |\n | NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS |\n | SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. |\n | |\n +-----------------------------------------------------------------------------*/\nvar ems = require('ems')(parseInt(process.argv[2]), false);\nvar assert = require('assert');\n\nvar data = ems.new(100, 10000, '/tmp/EMS_sanity');\n//data.writeXF(ems.myID, \"init\"+ems.myID)\ndata.writeXF(ems.myID, 100 + ems.myID);\nvar readback = data.readFF(ems.myID);\nvar computed = \"init\" + ems.myID;\nassert(data.readFF(ems.myID) === 100 + ems.myID, \"Readback FF was wrong: \" + readback + \" != \" + computed);\n\nems.barrier();\nassert(data.readFE(ems.myID) === 100 + (ems.myID) % ems.nThreads, \"Bad wrap one \" + data.read(ems.myID) + \" \" +\n (100 + ((ems.myID) % ems.nThreads)));\n\ndata.writeEF((ems.myID + 2) % ems.nThreads, \"second\" + ems.myID);\n\nreadback = data.readFF(ems.myID);\ncomputed = \"second\" + (ems.myID - 2 + ems.nThreads) % ems.nThreads;\nassert(readback === computed, \"Bad snake shift: \" + readback + \" != \" + computed);\n"
},
{
"path": "Tests/testUtils.js",
"content": "/*-----------------------------------------------------------------------------+\n | Extended Memory Semantics (EMS) Version 1.0.0 |\n | Synthetic Semantics http://www.synsem.com/ mogill@synsem.com |\n +-----------------------------------------------------------------------------+\n | Copyright (c) 2011-2014, Synthetic Semantics LLC. All rights reserved. |\n | Copyright (c) 2015-2016, Jace A Mogill. All rights reserved. |\n | |\n | Redistribution and use in source and binary forms, with or without |\n | modification, are permitted provided that the following conditions are met: |\n | * Redistributions of source code must retain the above copyright |\n | notice, this list of conditions and the following disclaimer. |\n | * Redistributions in binary form must reproduce the above copyright |\n | notice, this list of conditions and the following disclaimer in the |\n | documentation and/or other materials provided with the distribution. |\n | * Neither the name of the Synthetic Semantics nor the names of its |\n | contributors may be used to endorse or promote products derived |\n | from this software without specific prior written permission. |\n | |\n | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS |\n | \"AS IS\" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT |\n | LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR |\n | A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL SYNTHETIC |\n | SEMANTICS LLC BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, |\n | EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, |\n | PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR |\n | PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF |\n | LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING |\n | NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS |\n | SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. |\n | |\n +-----------------------------------------------------------------------------*/\n\nmodule.exports = {\n //-------------------------------------------------------------------\n // Timer functions\n timerStart: function () {\n return new Date().getTime();\n },\n timerStop: function (timer, nOps, label, myID) {\n var now = new Date().getTime();\n var opsPerSec = (nOps * 1000000) / ((now - timer) * 1000);\n if (typeof myID === undefined || myID === 0) {\n console.log(fmtNumber(nOps) + label + fmtNumber(Math.floor(opsPerSec).toString()) + \" ops/sec\");\n }\n },\n\n //---------------------------------------------------------------------------\n // Generate a random integer within a range (inclusive) from 'low' to 'high'\n randomInRange: function (low, high) {\n return ( Math.floor((Math.random() * (high - low)) + low) );\n }\n};\n\n\nfunction fmtNumber(n) {\n var s = ' ' + n.toString().replace(/\\B(?=(\\d{3})+(?!\\d))/g, \",\");\n if (n < 1) return n;\n else {\n return s.substr(s.length - 15, s.length);\n }\n}\n\n\n//---------------------------------------------------------------------------\n// Return an array with only unique elements\nArray.prototype.unique = function () {\n return this.filter(function (value, index, self) {\n return self.indexOf(value) === index;\n });\n};\n\n//---------------------------------------------------------------------------\n// Randomize the order of data in an array\nArray.prototype.shuffle = function () {\n for (var j, x, i = this.length; i; j = Math.floor(Math.random() * i), x = this[--i], this[i] = this[j], this[j] = x);\n return this;\n};\n\n"
},
{
"path": "Tests/test_alloc.c",
"content": "/*-----------------------------------------------------------------------------+\n | Extended Memory Semantics (EMS) Version 1.0.0 |\n | Synthetic Semantics http://www.synsem.com/ mogill@synsem.com |\n +-----------------------------------------------------------------------------+\n | Copyright (c) 2011-2014, Synthetic Semantics LLC. All rights reserved. |\n | Copyright (c) 2015-2016, Jace A Mogill. All rights reserved. |\n | |\n | Redistribution and use in source and binary forms, with or without |\n | modification, are permitted provided that the following conditions are met: |\n | * Redistributions of source code must retain the above copyright |\n | notice, this list of conditions and the following disclaimer. |\n | * Redistributions in binary form must reproduce the above copyright |\n | notice, this list of conditions and the following disclaimer in the |\n | documentation and/or other materials provided with the distribution. |\n | * Neither the name of the Synthetic Semantics nor the names of its |\n | contributors may be used to endorse or promote products derived |\n | from this software without specific prior written permission. |\n | |\n | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS |\n | \"AS IS\" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT |\n | LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR |\n | A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL SYNTHETIC |\n | SEMANTICS LLC BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, |\n | EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, |\n | PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR |\n | PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF |\n | LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING |\n | NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS |\n | SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. |\n | |\n +-----------------------------------------------------------------------------*/\n#include \"../Addon/ems_alloc.h\"\n#include \n\n\nstatic void\ntest_size(struct emsMem *b, int64_t addr) {\n int64_t s = emsMem_size(b,addr);\n printf(\"size %lld (sz = %lld)\\n\",addr,s);\n}\n\n\nstatic int64_t\ntest_alloc(struct emsMem *b, int64_t sz) {\n int64_t r = emsMem_alloc(b,sz);\n printf(\"alloc %lld (sz= %lld)\\n\",r,sz);\n //\temsMem_dump(b);\n test_size(b, r);\n return r;\n}\n\nstatic void\ntest_free(struct emsMem *b, int64_t addr) {\n printf(\"free %lld\\n\",addr);\n emsMem_free(b,addr);\n // emsMem_dump(b);\n}\n\n\n\nint main() {\n#define P(x) printf(\">>>>>>>%d %d\\n\", x, __builtin_ctzl(emsNextPow2(x)))\n /*\n P(1);\n P(10);\n P(127);\n P(128);\n P(129);\n exit(1);\n */\n\n struct emsMem * b = emsMem_new(16);\n\n // int64_t zz = test_alloc(b,65536);\n // printf(\"should fail: %lld\\n\", zz);\n\n int64_t x1 = test_alloc(b,65535);\n // int64_t x2 = test_alloc(b,1);\n emsMem_dump(b);\n\n test_free(b,x1);\n // test_free(b,x2);\n emsMem_dump(b);\n\n int64_t m1 = test_alloc(b,4);\n test_size(b,m1);\n int64_t m2 = test_alloc(b,9);\n test_size(b,m2);\n int64_t m3 = test_alloc(b,3);\n // struct emsMem *arr[100];\n int64_t arr[100];\n for(int i = 0; i < 50; i ++) {\n arr[i] = test_alloc(b, i*2);\n }\n test_size(b,m3);\n int64_t m4 = test_alloc(b,7);\n \n int64_t m5a = test_alloc(b,302);\n \n test_free(b,m3);\n test_free(b,m1);\n for(int64_t i = 0; i < 50; i ++) {\n test_free(b, arr[(i+13)%50]);\n }\n //\ttest_alloc(b, 1000);\n test_free(b,m4);\n test_free(b,m2);\n \n int64_t m5 = test_alloc(b,32);\n test_free(b,m5);\n \n int64_t m6 = test_alloc(b,0);\n test_free(b,m6);\n test_free(b,m5a); \n emsMem_dump(b);\n\n emsMem_delete(b);\n return 0;\n}\n"
},
{
"path": "Tests/tm.js",
"content": "/*-----------------------------------------------------------------------------+\n | Extended Memory Semantics (EMS) Version 1.0.0 |\n | Synthetic Semantics http://www.synsem.com/ mogill@synsem.com |\n +-----------------------------------------------------------------------------+\n | Copyright (c) 2011-2014, Synthetic Semantics LLC. All rights reserved. |\n | Copyright (c) 2015-2016, Jace A Mogill. All rights reserved. |\n | |\n | Redistribution and use in source and binary forms, with or without |\n | modification, are permitted provided that the following conditions are met: |\n | * Redistributions of source code must retain the above copyright |\n | notice, this list of conditions and the following disclaimer. |\n | * Redistributions in binary form must reproduce the above copyright |\n | notice, this list of conditions and the following disclaimer in the |\n | documentation and/or other materials provided with the distribution. |\n | * Neither the name of the Synthetic Semantics nor the names of its |\n | contributors may be used to endorse or promote products derived |\n | from this software without specific prior written permission. |\n | |\n | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS |\n | \"AS IS\" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT |\n | LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR |\n | A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL SYNTHETIC |\n | SEMANTICS LLC BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, |\n | EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, |\n | PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR |\n | PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF |\n | LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING |\n | NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS |\n | SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. |\n | |\n +-----------------------------------------------------------------------------*/\nvar ems = require('ems')(parseInt(process.argv[2]));\nvar util = require('./testUtils');\nvar assert = require('assert');\nvar arrLen = 1000000;\nvar nTransactions = 1000000;\nvar nTables = 6;\nvar maxNops = 5;\nvar tables = [];\nvar workQ = ems.new({\n dimensions: [nTransactions + ems.nThreads],\n heapSize: nTransactions * 200,\n useExisting: false\n});\n\nvar totalNops = ems.new(2);\nvar checkNops = ems.new(1);\n\n\nvar startTime = util.timerStart();\n//---------------------------------------------------------------------------\n// Create all the tables\nfor (var tableN = 0; tableN < nTables; tableN++) {\n tables[tableN] = ems.new({\n dimensions: [arrLen],\n heapSize: 0,\n useExisting: false,\n filename: '/tmp/EMS_tm' + tableN,\n dataFill: 0,\n doDataFill: true\n });\n}\nutil.timerStop(startTime, nTables, \" tables initialized \", ems.myID);\n\n\n//---------------------------------------------------------------------------\n// One thread enqueues all the work\nems.master(function () {\n totalNops.writeXF(0, 0);\n totalNops.writeXF(1, 0);\n checkNops.writeXF(0, 0);\n});\nems.barrier();\n\n\nstartTime = util.timerStart();\n//---------------------------------------------------------------------------\n// Generate operations involving random elements in random EMS arrays\n// and enqueue them on the work queue\nems.parForEach(0, nTransactions, function (transN) {\n var ops = [];\n var nOps = util.randomInRange(1, maxNops);\n for (var opN = 0; opN < nOps; opN++) {\n var tableN = util.randomInRange(0, nTables);\n var idx = util.randomInRange(0, arrLen);\n if (transN % 2 == 0 || opN % 3 > 0) {\n ops.push([tableN, idx, true]);\n }\n//\tif(opN % 3 > 0) { ops.push([tableN, idx, true]) }\n else {\n ops.push([tableN, idx]);\n }\n }\n\n\n // De-duplicate operations in a transaction which would deadlock\n var indicies = [];\n var uids = [];\n for (var i = 0; i < ops.length; i++) {\n indicies[i] = i;\n uids[i] = (ops[i][0] * 1000000000000) + ops[i][1];\n }\n var uniq = [];\n for (opN = 0; opN < ops.length; opN++) {\n var isDupl = false;\n for (var checkN = 0; checkN < ops.length; checkN++) {\n if (opN != checkN && uids[opN] == uids[checkN]) {\n isDupl = true;\n break;\n }\n }\n if (!isDupl) {\n uniq.push(ops[opN]);\n }\n }\n workQ.enqueue(JSON.stringify(uniq));\n});\n\n// After all the work has been enqueued, add DONE semaphores to the\n// end of the queue so they are processed only after all the work\n// has been issued. Each thread enqueues one event and can only\n// consume one before exiting.\nworkQ.enqueue(\"DONE\");\nutil.timerStop(startTime, nTransactions, \" transactions enqueued \", ems.myID);\n\n\nvar rwNops = 0;\nvar readNops = 0;\nstartTime = util.timerStart();\n\nwhile (true) {\n var str = workQ.dequeue();\n if (str !== undefined) {\n if (str === \"DONE\") {\n break;\n } else {\n var ops = JSON.parse(str);\n for (var opN = 0; opN < ops.length; opN++) {\n ops[opN][0] = tables[ops[opN][0]];\n }\n var transaction = ems.tmStart(ops);\n ops.forEach(function (op) {\n var tmp = op[0].read(op[1]);\n if (op[2] != true) {\n rwNops++;\n op[0].write(op[1], tmp + 1);\n } else {\n readNops++;\n }\n });\n ems.tmEnd(transaction, true);\n }\n } else {\n ems.diag(\"Should not find nothing in queue\");\n }\n}\ntotalNops.faa(0, rwNops);\ntotalNops.faa(1, readNops);\n\nems.barrier();\nutil.timerStop(startTime, nTransactions, \" transactions performed\", ems.myID);\nutil.timerStop(startTime, totalNops.readFF(0), \" table updates \", ems.myID);\nutil.timerStop(startTime, totalNops.readFF(0) + totalNops.readFF(1), \" elements referenced \", ems.myID);\n\nstartTime = util.timerStart();\nems.parForEach(0, nTables, function (tableN) {\n var localSum = 0;\n for (var idx = 0; idx < arrLen; idx++) {\n localSum += tables[tableN].read(idx);\n }\n checkNops.faa(0, localSum);\n}, 'dynamic');\nutil.timerStop(startTime, nTables * arrLen, \" elements checked \", ems.myID);\n\nems.master(function () {\n assert(checkNops.readFF(0) === totalNops.readFF(0),\n \"Error in final sum = \" + checkNops.readFF(0) + \" should be=\" + totalNops.readFF(0));\n});\n"
},
{
"path": "Tests/tm_noq.js",
"content": "/*-----------------------------------------------------------------------------+\n | Extended Memory Semantics (EMS) Version 1.3.0 |\n | Synthetic Semantics http://www.synsem.com/ mogill@synsem.com |\n +-----------------------------------------------------------------------------+\n | Copyright (c) 2011-2014, Synthetic Semantics LLC. All rights reserved. |\n | Copyright (c) 2015-2016, Jace A Mogill. All rights reserved. |\n | |\n | Redistribution and use in source and binary forms, with or without |\n | modification, are permitted provided that the following conditions are met: |\n | * Redistributions of source code must retain the above copyright |\n | notice, this list of conditions and the following disclaimer. |\n | * Redistributions in binary form must reproduce the above copyright |\n | notice, this list of conditions and the following disclaimer in the |\n | documentation and/or other materials provided with the distribution. |\n | * Neither the name of the Synthetic Semantics nor the names of its |\n | contributors may be used to endorse or promote products derived |\n | from this software without specific prior written permission. |\n | |\n | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS |\n | \"AS IS\" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT |\n | LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR |\n | A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL SYNTHETIC |\n | SEMANTICS LLC BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, |\n | EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, |\n | PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR |\n | PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF |\n | LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING |\n | NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS |\n | SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. |\n | |\n +-----------------------------------------------------------------------------*/\nvar ems = require('ems')(parseInt(process.argv[2]));\nvar util = require('./testUtils');\nvar assert = require('assert');\nvar arrLen = 1000000;\nvar nTransactions = 400000;\nvar nTables = 6;\nvar maxNops = 5;\nvar tables = [];\nvar totalNops = ems.new(2);\nvar checkNops = ems.new(1);\n\n//---------------------------------------------------------------------------\n// Create all the tables\nfor (var tableN = 0; tableN < nTables; tableN++) {\n tables[tableN] = ems.new({\n dimensions: [arrLen],\n heapSize: 0,\n useExisting: false,\n filename: '/tmp/EMS_tm' + tableN,\n dataFill: 0,\n doDataFill: true,\n setFEtags: 'full'\n });\n}\n\n\n// One thread enqueues all the work\nems.master(function () {\n totalNops.writeXF(0, 0);\n totalNops.writeXF(1, 0);\n checkNops.writeXF(0, 0);\n});\nems.barrier();\n\nfunction makeWork() {\n var ops = [];\n var nOps = util.randomInRange(1, maxNops);\n for (var opN = 0; opN < nOps; opN++) {\n var tableN = util.randomInRange(0, nTables);\n var idx = util.randomInRange(0, arrLen);\n if (idx % 10 < 5 || opN % 3 > 0) {\n ops.push([tableN, idx, true]);\n } else {\n ops.push([tableN, idx]);\n }\n }\n\n // De-duplicate operations in a transaction which would deadlock\n var indicies = [];\n var uids = [];\n for (var i = 0; i < ops.length; ++i) {\n indicies[i] = i;\n uids[i] = (ops[i][0] * 1000000000000) + ops[i][1];\n }\n var uniq = [];\n for (opN = 0; opN < ops.length; opN++) {\n var isDupl = false;\n for (var checkN = 0; checkN < ops.length; checkN++) {\n if (opN != checkN && uids[opN] == uids[checkN]) {\n isDupl = true;\n break;\n }\n }\n if (!isDupl) {\n uniq.push(ops[opN]);\n }\n }\n return (JSON.stringify(uniq));\n}\n\n\nvar rwNops = 0;\nvar readNops = 0;\nvar startTime = util.timerStart();\n\nems.parForEach(0, nTransactions, function (iter) {\n var str = makeWork(iter);\n if (str !== undefined) {\n var ops = JSON.parse(str);\n for (var opN = 0; opN < ops.length; opN++) {\n ops[opN][0] = tables[ops[opN][0]];\n }\n var transaction = ems.tmStart(ops);\n ops.forEach(function (op) {\n var tmp = op[0].read(op[1]);\n if (op[2] != true) {\n rwNops++;\n op[0].write(op[1], tmp + 1);\n } else {\n readNops++;\n }\n });\n ems.tmEnd(transaction, true);\n }\n});\ntotalNops.faa(0, rwNops);\ntotalNops.faa(1, readNops);\nems.barrier();\nutil.timerStop(startTime, nTransactions, \" transactions performed \", ems.myID);\nutil.timerStop(startTime, totalNops.readFF(0), \" table updates \", ems.myID);\nutil.timerStop(startTime, totalNops.readFF(0) + totalNops.readFF(1), \" elements referenced \", ems.myID);\n\n\nstartTime = util.timerStart();\nems.parForEach(0, nTables, function (tableN) {\n var localSum = 0;\n for (var idx = 0; idx < arrLen; idx++) {\n localSum += tables[tableN].read(idx);\n }\n checkNops.faa(0, localSum);\n}, 'dynamic');\nutil.timerStop(startTime, nTables * arrLen, \" elements checked \", ems.myID);\n\n\nems.master(function () {\n assert(checkNops.readFF(0) == totalNops.readFF(0),\n \"Error in final sum = \" + checkNops.readFF(0) + \" should be=\" + totalNops.readFF(0));\n});\n"
},
{
"path": "Tests/tm_noq_strings.js",
"content": "/*-----------------------------------------------------------------------------+\n | Extended Memory Semantics (EMS) Version 1.4.0 |\n | Synthetic Semantics http://www.synsem.com/ mogill@synsem.com |\n +-----------------------------------------------------------------------------+\n | Copyright (c) 2011-2014, Synthetic Semantics LLC. All rights reserved. |\n | Copyright (c) 2015-2016, Jace A Mogill. All rights reserved. |\n | |\n | Redistribution and use in source and binary forms, with or without |\n | modification, are permitted provided that the following conditions are met: |\n | * Redistributions of source code must retain the above copyright |\n | notice, this list of conditions and the following disclaimer. |\n | * Redistributions in binary form must reproduce the above copyright |\n | notice, this list of conditions and the following disclaimer in the |\n | documentation and/or other materials provided with the distribution. |\n | * Neither the name of the Synthetic Semantics nor the names of its |\n | contributors may be used to endorse or promote products derived |\n | from this software without specific prior written permission. |\n | |\n | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS |\n | \"AS IS\" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT |\n | LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR |\n | A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL SYNTHETIC |\n | SEMANTICS LLC BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, |\n | EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, |\n | PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR |\n | PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF |\n | LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING |\n | NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS |\n | SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. |\n | |\n +-----------------------------------------------------------------------------*/\nvar ems = require('ems')(parseInt(process.argv[2]));\nvar util = require('./testUtils');\nvar arrLen = 1000000;\nvar nTransactions = 4000000;\nvar nTables = 6;\nvar maxNops = 5;\nvar tables = [];\nvar totalNops = ems.new(2);\nvar checkNops = ems.new(1);\nvar tableN;\n\n\n//---------------------------------------------------------------------------\n// Create all the tables\nfor (tableN = 0; tableN < nTables; tableN++) {\n tables[tableN] = ems.new({\n dimensions: [arrLen],\n heapSize: arrLen * 40,\n useExisting: false,\n filename: '/tmp/EMS_tm' + tableN,\n dataFill: 0,\n doDataFill: true\n });\n}\n\n\n// One thread enqueues all the work\nems.master(function () {\n totalNops.writeXF(0, 0);\n totalNops.writeXF(1, 0);\n checkNops.writeXF(0, 0);\n});\nems.barrier();\n\nfunction makeWork() {\n var ops = [];\n var nOps = util.randomInRange(1, maxNops);\n // var indexes = []\n for (var opN = 0; opN < nOps; opN++) {\n var tableN = util.randomInRange(0, nTables);\n var idx = util.randomInRange(0, arrLen);\n if (idx % 10 < 5 || opN % 3 > 0) {\n ops.push([tableN, idx, true]);\n }\n else {\n ops.push([tableN, idx]);\n }\n }\n\n // De-duplicate operations in a transaction which would deadlock\n var indicies = [];\n var uids = [];\n for (var i = 0; i < ops.length; ++i) {\n indicies[i] = i;\n uids[i] = (ops[i][0] * 1000000000000) + ops[i][1];\n }\n var uniq = [];\n for (opN = 0; opN < ops.length; opN++) {\n var isDupl = false;\n for (var checkN = 0; checkN < ops.length; checkN++) {\n if (opN != checkN && uids[opN] == uids[checkN]) {\n isDupl = true;\n break;\n }\n }\n if (!isDupl) {\n uniq.push(ops[opN]);\n }\n }\n return (JSON.stringify(uniq));\n}\n\n\nvar rwNops = 0;\nvar readNops = 0;\nvar startTime = util.timerStart();\n\n//---------------------------------------------------------------------------\n// Generate operations involving random elements in random EMS arrays\n// and enqueue them on the work queue\n//\nems.parForEach(0, nTransactions, function (iter) {\n var str = makeWork(iter);\n if (str !== undefined) {\n var ops = JSON.parse(str);\n for (var opN = 0; opN < ops.length; opN++) {\n ops[opN][0] = tables[ops[opN][0]];\n }\n var transaction = ems.tmStart(ops);\n ops.forEach(function (op) {\n var tmp = op[0].read(op[1]);\n if (op[2] != true) {\n rwNops++;\n op[0].write(op[1], tmp + ' ' + 1);\n } else {\n readNops++;\n }\n });\n ems.tmEnd(transaction, true);\n }\n});\ntotalNops.faa(0, rwNops);\ntotalNops.faa(1, readNops);\n\nems.barrier();\nutil.timerStop(startTime, nTransactions, \" transactions performed \", ems.myID);\nutil.timerStop(startTime, totalNops.readFF(0), \" table updates \", ems.myID);\nutil.timerStop(startTime, totalNops.readFF(0) + totalNops.readFF(1), \" elements referenced \", ems.myID);\n\nstartTime = util.timerStart();\nif (true) {\n ems.parForEach(0, nTables, function (tableN) {\n // var localSum = 0\n for (var idx = 0; idx < arrLen; idx++) {\n tables[tableN].read(idx);\n }\n }, 'dynamic');\n} else {\n for (tableN = 0; tableN < nTables; tableN++) {\n ems.parForEach(0, arrLen, function (idx) {\n checkNops.faa(0, tables[tableN].read(idx));\n });\n }\n}\nutil.timerStop(startTime, nTables * arrLen, \" elements checked \", ems.myID);\n"
},
{
"path": "Tests/v8Types.js",
"content": "/*-----------------------------------------------------------------------------+\n | Extended Memory Semantics (EMS) Version 1.3.0 |\n | Synthetic Semantics http://www.synsem.com/ mogill@synsem.com |\n +-----------------------------------------------------------------------------+\n | Copyright (c) 2011-2014, Synthetic Semantics LLC. All rights reserved. |\n | Copyright (c) 2015-2016, Jace A Mogill. All rights reserved. |\n | |\n | Redistribution and use in source and binary forms, with or without |\n | modification, are permitted provided that the following conditions are met: |\n | * Redistributions of source code must retain the above copyright |\n | notice, this list of conditions and the following disclaimer. |\n | * Redistributions in binary form must reproduce the above copyright |\n | notice, this list of conditions and the following disclaimer in the |\n | documentation and/or other materials provided with the distribution. |\n | * Neither the name of the Synthetic Semantics nor the names of its |\n | contributors may be used to endorse or promote products derived |\n | from this software without specific prior written permission. |\n | |\n | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS |\n | \"AS IS\" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT |\n | LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR |\n | A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL SYNTHETIC |\n | SEMANTICS LLC BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, |\n | EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, |\n | PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR |\n | PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF |\n | LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING |\n | NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS |\n | SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. |\n | |\n +-----------------------------------------------------------------------------*/\n'use strict';\nvar ems = require('ems')(parseInt(process.argv[2]), false);\nvar assert = require('assert');\nvar arrLen = 10000;\nvar a = ems.new(arrLen, arrLen * 400);\n\nvar arrayElem = ['abcd', true, 1234.567, false, {x: 'xxx', y: 'yyyyy'}, 987, null, [10, 11, 12, 13]];\nvar objElem = {a: 1, b: 321.653, c: 'asdasd'};\n\nvar objMap = ems.new({\n dimensions: [arrLen],\n heapSize: arrLen * 200,\n useMap: true,\n useExisting: false,\n setFEtags: 'full'\n});\n\n\nvar arrObj = ems.new({\n dimensions: [arrLen],\n heapSize: arrLen * 200,\n useMap: false,\n useExisting: false,\n setFEtags: 'full',\n dataFill: arrayElem,\n doDataFill: true\n});\n\n\nif (ems.myID == 0) {\n objMap.writeXF('any obj', objElem);\n assert(objMap.readFF('any obj').a === objElem.a);\n assert(objMap.readFF('any obj').b === objElem.b);\n assert(objMap.readFE('any obj').c === objElem.c);\n\n arrayElem.forEach(function (elem, idx) {\n if (typeof elem === 'object') {\n assert(typeof arrObj.readFF(123)[idx] === 'object');\n } else {\n var readback = arrObj.readFF(123);\n assert(readback[idx] === elem);\n }\n });\n arrObj.readFE(123);\n\n\n var newObj = {xa: 10000, xb: 32100.653, xc: 'xxxxxxxasdasd'};\n objMap.writeEF('any obj', newObj);\n assert(objMap.readFF('any obj').xa === newObj.xa);\n assert(objMap.readFF('any obj').xb === newObj.xb);\n assert(objMap.readFE('any obj').xc === newObj.xc);\n\n arrObj.writeEF(123, 'overwrite the old array');\n arrObj.writeXF(1, []);\n var newArr = [9, 8, 7, 6, , , , 'abs', {one: 1, two: 2, three: 'threeeeee'}, 1, 2, 3];\n arrObj.writeXF(2, newArr);\n\n assert(arrObj.readFE(123) === 'overwrite the old array');\n assert(arrObj.readFE(1).length === 0);\n newArr.forEach(function (elem, idx) {\n if (typeof elem == 'object') {\n if (typeof arrObj.readFF(2)[idx] != 'object') {\n console.log('Object in array is no longer an object?');\n }\n } else {\n assert(arrObj.readFF(2)[idx] === elem);\n }\n });\n\n var newerObj = {q: 123, r: 345, x: [1, 2, 3, 4]};\n arrObj.writeEF(123, newerObj);\n assert(arrObj.readFF(123).q === newerObj.q);\n assert(arrObj.readFF(123).r === newerObj.r);\n assert(arrObj.readFF(123).x[2] === newerObj.x[2]);\n\n}\n\nems.barrier();\n\n//----------------------------------------\n\nvar newIdx, newVal, oldVal, js;\nvar data = [false, true, 1234, 987.654321, 'hello', undefined];\n\nfor (var old = 0; old < data.length; old++) {\n for (newIdx = 0; newIdx < data.length; newIdx++) {\n a.writeXF(ems.myID, data[old]);\n js = data[old];\n js += data[newIdx];\n oldVal = a.faa(ems.myID, data[newIdx]);\n newVal = a.readFF(ems.myID);\n\n assert(((newVal === js) || (isNaN(oldVal) && isNaN(newVal) ) ||\n (isNaN(newVal) && data[newIdx] === undefined) ),\n 'FAA: old=' + data[old] + ' new=' + data[newIdx] +\n ' oldVal=' + oldVal + '/' + typeof oldVal + ' newVal=' +\n newVal + '/' + typeof newVal + ' js=' + js + '/' + typeof js);\n }\n}\n\n\nems.barrier();\nvar id = (ems.myID + 1) % ems.nThreads;\nfor (var memIdx = 0; memIdx < data.length; memIdx++) {\n for (var oldIdx = 0; oldIdx < data.length; oldIdx++) {\n for (newIdx = 0; newIdx < data.length; newIdx++) {\n a.writeXF(id, data[memIdx]);\n var memVal = a.cas(id, data[oldIdx], data[newIdx]);\n newVal = a.readFF(id);\n js = data[memIdx];\n if (js === data[oldIdx]) {\n js = data[newIdx];\n }\n\n assert(js === newVal,\n 'CAS: intial=' + data[memIdx] + \" memval=\" + memVal +\n \" readback=\" + newVal + \" oldVal=\" + data[oldIdx] +\n ' expected=' + data[newIdx] + ' JS:' + js);\n }\n }\n}\n\n\nems.parForEach(0, arrLen, function (idx) {\n a.writeXF(idx, undefined);\n a.faa(idx, undefined);\n a.faa(idx, 'bye byte');\n a.faa(idx, ems.myID);\n a.faa(idx, 0.1);\n a.faa(idx, false);\n assert(a.readFF(idx) == 'nanbye byte' + ems.myID + '0.100000false',\n 'Failed match =' + a.read(idx));\n});\n\n\n//-----------------------------------------------------------\n\n\nems.parForEach(0, arrLen, function (idx) {\n a.writeXF(idx, 0);\n});\n\nvar nTimes = 150;\nems.parForEach(0, arrLen, function (previdx) {\n for (var i = 0; i < nTimes; i++) {\n a.faa((previdx + i) % arrLen, 1)\n }\n});\n\nems.parForEach(0, arrLen, function (idx) {\n a.faa(idx, 0.1);\n a.faa(idx, 'fun!');\n a.faa(idx, false);\n assert(a.readFE(idx) == nTimes + '.100000fun!false',\n 'Failed match =' + a.read(idx) + ' idx=' + idx);\n });\n"
},
{
"path": "Vagrantfile",
"content": "# -*- mode: ruby -*-\n# vi: set ft=ruby :\n\n# All Vagrant configuration is done below. The \"2\" in Vagrant.configure\n# configures the configuration version (we support older styles for\n# backwards compatibility). Please don't change it unless you know what\n# you're doing.\nVagrant.configure(2) do |config|\n # The most common configuration options are documented and commented below.\n # For a complete reference, please see the online documentation at\n # https://docs.vagrantup.com.\n\n # Every Vagrant development environment requires a box. You can search for\n # boxes at https://atlas.hashicorp.com/search.\n config.vm.box = \"ubuntu/trusty64\"\n\n # Disable automatic box update checking. If you disable this, then\n # boxes will only be checked for updates when the user runs\n # `vagrant box outdated`. This is not recommended.\n # config.vm.box_check_update = false\n\n # Create a forwarded port mapping which allows access to a specific port\n # within the machine from a port on the host machine. In the example below,\n # accessing \"localhost:8080\" will access port 80 on the guest machine.\n # config.vm.network \"forwarded_port\", guest: 80, host: 8080\n\n # Create a private network, which allows host-only access to the machine\n # using a specific IP.\n # config.vm.network \"private_network\", ip: \"192.168.33.10\"\n\n # Create a public network, which generally matched to bridged network.\n # Bridged networks make the machine appear as another physical device on\n # your network.\n # config.vm.network \"public_network\"\n\n # Share an additional folder to the guest VM. The first argument is\n # the path on the host to the actual folder. The second argument is\n # the path on the guest to mount the folder. And the optional third\n # argument is a set of non-required options.\n # config.vm.synced_folder \"../data\", \"/vagrant_data\"\n\n # Provider-specific configuration so you can fine-tune various\n # backing providers for Vagrant. These expose provider-specific options.\n # Example for VirtualBox:\n #\n # config.vm.provider \"virtualbox\" do |vb|\n # # Display the VirtualBox GUI when booting the machine\n # vb.gui = true\n #\n # # Customize the amount of memory on the VM:\n # vb.memory = 1024\n # vb.cpus = 1\n # end\n #\n # View the documentation for the provider you are using for more\n # information on available options.\n\n # Define a Vagrant Push strategy for pushing to Atlas. Other push strategies\n # such as FTP and Heroku are also available. See the documentation at\n # https://docs.vagrantup.com/v2/push/atlas.html for more information.\n # config.push.define \"atlas\" do |push|\n # push.app = \"YOUR_ATLAS_USERNAME/YOUR_APPLICATION_NAME\"\n # end\n\n # Enable provisioning with a shell script. Additional provisioners such as\n # Puppet, Chef, Ansible, Salt, and Docker are also available. Please see the\n # documentation for more information about their specific syntax and use.\n # config.vm.provision \"shell\", inline: <<-SHELL\n # sudo apt-get update\n # sudo apt-get install -y apache2\n # SHELL\n\n config.vm.provision \"shell\", inline: <<-SHELL\n sudo apt-get update\n sudo curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.35.3/install.sh | bash\n SHELL\n\nend\n"
},
{
"path": "binding.gyp",
"content": "{\n \"targets\": [\n {\n \"target_name\": \"ems\",\n \"sources\": [\n \"src/collectives.cc\", \"src/ems.cc\", \"src/ems_alloc.cc\", \"src/loops.cc\",\n \"nodejs/nodejs.cc\", \"src/primitives.cc\", \"src/rmw.cc\"],\n 'include_dirs': [\"\n\n\n \n\n\n\n"
},
{
"path": "nodejs/ems.js",
"content": "/*-----------------------------------------------------------------------------+\n | Extended Memory Semantics (EMS) Version 1.6.1 |\n | http://mogill.com/ jace@mogill.com |\n +-----------------------------------------------------------------------------+\n | Copyright (c) 2011-2014, Synthetic Semantics LLC. All rights reserved. |\n | Copyright (c) 2019 Aleksander J Budzynowski. Update NAN to N-API. |\n | Copyright (c) 2015-2020, Jace A Mogill. All rights reserved. |\n | |\n | Redistribution and use in source and binary forms, with or without |\n | modification, are permitted provided that the following conditions are met: |\n | * Redistributions of source code must retain the above copyright |\n | notice, this list of conditions and the following disclaimer. |\n | * Redistributions in binary form must reproduce the above copyright |\n | notice, this list of conditions and the following disclaimer in the |\n | documentation and/or other materials provided with the distribution. |\n | * Neither the name of the Synthetic Semantics nor the names of its |\n | contributors may be used to endorse or promote products derived |\n | from this software without specific prior written permission. |\n | |\n | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS |\n | \"AS IS\" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT |\n | LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR |\n | A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL SYNTHETIC |\n | SEMANTICS LLC BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, |\n | EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, |\n | PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR |\n | PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF |\n | LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING |\n | NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS |\n | SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. |\n | |\n +-----------------------------------------------------------------------------*/\n\"use strict\";\nvar fs = require(\"fs\");\nvar child_process = require(\"child_process\");\nvar EMS = require(\"bindings\")(\"ems.node\");\nvar EMSglobal;\n\n// The Proxy object is built in or defined by Reflect\ntry {\n var EMS_Harmony_Reflect = require(\"harmony-reflect\");\n} catch (err) {\n // Not installed, but possibly not needed anyhow\n}\n\n//==================================================================\n// Convert the different possible index types into a linear EMS index\nfunction EMSidx(indexes, // Index given by application\n EMSarray) { // EMS array being indexed\n var idx = 0;\n if (indexes instanceof Array) { // Is a Multidimension array: [x,y,z]\n indexes.forEach(function (x, i) {\n idx += x * EMSarray.dimStride[i];\n });\n } else {\n if (!(typeof indexes === \"number\") && !EMSarray.useMap) { // If no map, only use integers\n console.log(\"EMS ERROR: Non-integer index used, but EMS memory was not configured to use a map (useMap)\",\n indexes, typeof indexes, EMSarray.useMap);\n idx = -1;\n } else { // Is a mappable intrinsic type\n idx = indexes;\n }\n }\n return idx;\n}\n\n\n//==================================================================\n// Print a message to the console with a prefix indicating which\n// thread is printing\nfunction EMSdiag(text) {\n console.log(\"EMStask \" + this.myID + \": \" + text);\n}\n\n\n//==================================================================\n// Co-Begin a parallel region, executing the function \"func\"\nfunction EMSparallel() {\n EMSglobal.inParallelContext = true;\n var user_args = (arguments.length === 1?[arguments[0]]:Array.apply(null, arguments));\n var func = user_args.pop(); // Remove the function\n // Loop over remote processes, starting each of them\n this.tasks.forEach(function (task, taskN) {\n task.send({\"taskN\": taskN + 1, \"args\": user_args, \"func\": func.toString()});\n task.send({\"taskN\": taskN + 1, \"args\": [], \"func\": \"function() { ems.barrier(); }\"});\n });\n func.apply(null, user_args); // Invoke on master process\n EMSbarrier(); // Wait for all processes to finish\n EMSglobal.inParallelContext = false;\n}\n\n\n//==================================================================\n// Execute the local iterations of a decomposed loop with\n// the specified scheduling.\nfunction EMSparForEach(start, // First iteration's index\n end, // Final iteration's index\n loopBody, // Function to execute each iteration\n scheduleType, // Load balancing technique\n minChunk) { // Smallest block of iterations\n var sched = scheduleType;\n var idx;\n if (typeof sched === \"undefined\") {\n sched = \"guided\";\n }\n if (typeof minChunk === \"undefined\") {\n minChunk = 1;\n }\n switch (scheduleType) {\n case \"static\": // Evenly divide the iterations across threads\n var range = end - start;\n var blocksz = Math.floor(range / EMSglobal.nThreads) + 1;\n var s = blocksz * EMSglobal.myID;\n var e = blocksz * (EMSglobal.myID + 1);\n if (e > end) {\n e = end;\n }\n for (idx = s; idx < e; idx += 1) {\n loopBody(idx + start);\n }\n break;\n case \"dynamic\":\n case \"guided\":\n default:\n // Initialize loop bounds, block size, etc.\n EMSglobal.loopInit(start, end, sched, minChunk);\n // Do not enter loop until all threads have completed initialization\n // If the barrier is present at the loop end, this may replaced\n // with first-thread initialization.\n var extents;\n EMSbarrier();\n do {\n extents = EMSglobal.loopChunk();\n for (idx = extents.start; idx < extents.end; idx++) {\n loopBody(idx);\n }\n } while (extents.end - extents.start > 0);\n }\n\n // Do not proceed until all iterations have completed\n // TODO: OpenMP offers NOWAIT to skip this barrier\n EMSbarrier();\n}\n\n\n//==================================================================\n// Start a Transaction\n// Input is an array containing EMS elements to transition from\n// Full to Empty:\n// arr = [ [ emsArr0, idx0 ], [ emsArr1, idx1, true ], [ emsArr2, idx2 ] ]\nfunction EMStmStart(emsElems) {\n var MAX_ELEM_PER_REGION = 10000000000000;\n\n // Build a list of indexes of the elements to lock which will hold\n // the elements sorted for deadlock free acquisition\n var indicies = [];\n for (var i = 0; i < emsElems.length; ++i) indicies[i] = i;\n\n // Sort the elements to lock according to a global ordering\n var sorted = indicies.sort(function (a, b) {\n return ( ((emsElems[a][0].regionN * MAX_ELEM_PER_REGION) + emsElems[a][1]) -\n ((emsElems[b][0].regionN * MAX_ELEM_PER_REGION) + emsElems[b][1]) )\n });\n\n // Acquire the locks in the deadlock free order, saving the contents\n // of the memory when it locked.\n // Mark read-write data as Empty, and read-only data under a readers-writer lock\n var tmHandle = [];\n sorted.forEach(function (e) {\n var val;\n if (emsElems[e][2] === true) {\n val = emsElems[e][0].readRW(emsElems[e][1]);\n } else {\n val = emsElems[e][0].readFE(emsElems[e][1]);\n }\n tmHandle.push([emsElems[e][0], emsElems[e][1], emsElems[e][2], val])\n });\n return tmHandle;\n}\n\n\n//==================================================================\n// Commit or abort a transaction\n// The tmHandle contains the result from tmStart:\n// [ [ EMSarray, index, isReadOnly, origValue ], ... ]\nfunction EMStmEnd(tmHandle, // The returned value from tmStart\n doCommit) { // Commit or Abort the transaction\n tmHandle.forEach(function (emsElem) {\n if (emsElem[2] === true) {\n // Is a read-only element\n emsElem[0].releaseRW(emsElem[1]);\n } else {\n if (doCommit) {\n // Is read-write, keep current value and mark Full\n emsElem[0].setTag(emsElem[1], true);\n } else {\n // Write back the original value and mark full\n emsElem[0].writeEF(emsElem[1], emsElem[3]);\n }\n }\n });\n}\n\n\n//==================================================================\nfunction EMSreturnData(value) {\n if (typeof value === \"object\") {\n var retval;\n try {\n if (value.data[0] === \"[\" && value.data.slice(-1) === \"]\") {\n retval = eval(value.data);\n } else {\n retval = JSON.parse(value.data);\n }\n } catch (err) {\n // TODO: Throw error?\n console.log('EMSreturnData: Corrupt memory, unable to reconstruct JSON value of (' +\n value.data + ')\\n');\n retval = undefined;\n }\n return retval;\n } else {\n return value;\n }\n}\n\n\n//==================================================================\n// Synchronize memory with storage\n//\nfunction EMSsync(emsArr) {\n return this.data.sync(emsArr);\n}\n\n\n//==================================================================\n// Convert an EMS index into a mapped key\nfunction EMSindex2key(index) {\n if(typeof(index) !== \"number\") {\n console.log('EMSindex2key: Index (' + index + ') is not an integer');\n return undefined;\n }\n\n return this.data.index2key(index);\n}\n\n\n//==================================================================\n// Wrappers around Stacks and Queues\nfunction EMSpush(value) {\n if (typeof value === \"object\") {\n return this.data.push(JSON.stringify(value), true);\n } else {\n return this.data.push(value);\n }\n}\n\nfunction EMSpop() {\n return EMSreturnData(this.data.pop());\n}\n\nfunction EMSdequeue() {\n return EMSreturnData(this.data.dequeue());\n}\n\nfunction EMSenqueue(value) {\n if (typeof value === \"object\") {\n return this.data.enqueue(JSON.stringify(value), true); // Retuns only integers\n } else {\n return this.data.enqueue(value); // Retuns only integers\n }\n}\n\n\n//==================================================================\n// Wrappers around Primitive AMOs\n// Translate EMS maps and multi-dimensional array indexes/keys\n// into EMS linear addresses\n// Apparently it is illegal to pass a native function as an argument\nfunction EMSwrite(indexes, value) {\n var linearIndex = EMSidx(indexes, this);\n if (typeof value === \"object\") {\n this.data.write(linearIndex, JSON.stringify(value), true);\n } else {\n this.data.write(linearIndex, value);\n }\n}\n\nfunction EMSwriteEF(indexes, value) {\n var linearIndex = EMSidx(indexes, this);\n if (typeof value === \"object\") {\n this.data.writeEF(linearIndex, JSON.stringify(value), true);\n } else {\n this.data.writeEF(linearIndex, value);\n }\n}\n\nfunction EMSwriteXF(indexes, value) {\n var linearIndex = EMSidx(indexes, this);\n if (typeof value === \"object\") {\n this.data.writeXF(linearIndex, JSON.stringify(value), true);\n } else {\n this.data.writeXF(linearIndex, value);\n }\n}\n\nfunction EMSwriteXE(indexes, value) {\n var nativeIndex = EMSidx(indexes, this);\n if (typeof value === \"object\") {\n this.data.writeXE(nativeIndex, JSON.stringify(value), true);\n } else {\n this.data.writeXE(nativeIndex, value);\n }\n}\n\nfunction EMSread(indexes) {\n return EMSreturnData(this.data.read(EMSidx(indexes, this)))\n}\n\nfunction EMSreadFE(indexes) {\n return EMSreturnData(this.data.readFE(EMSidx(indexes, this)))\n}\n\nfunction EMSreadFF(indexes) {\n return EMSreturnData(this.data.readFF(EMSidx(indexes, this)))\n}\n\nfunction EMSreadRW(indexes) {\n return EMSreturnData(this.data.readRW(EMSidx(indexes, this)))\n}\n\nfunction EMSreleaseRW(indexes) {\n return this.data.releaseRW(EMSidx(indexes, this))\n}\n\nfunction EMSsetTag(indexes, fe) {\n return this.data.setTag(EMSidx(indexes, this), fe)\n}\n\nfunction EMSfaa(indexes, val) {\n if (typeof val === \"object\") {\n console.log(\"EMSfaa: Cannot add an object to something\");\n return undefined;\n } else {\n return this.data.faa(EMSidx(indexes, this), val); // No returnData(), FAA can only return JSON primitives\n }\n}\n\nfunction EMScas(indexes, oldVal, newVal) {\n if (typeof newVal === \"object\") {\n console.log(\"EMScas: ERROR -- objects are not a valid new type\");\n return undefined;\n } else {\n return this.data.cas(EMSidx(indexes, this), oldVal, newVal);\n }\n}\n\n\n//==================================================================\n// Serialize execution through this function\nfunction EMScritical(func, timeout) {\n if (typeof timeout === \"undefined\") {\n timeout = 500000; // TODO: Magic number -- long enough for errors, not load imbalance\n }\n EMSglobal.criticalEnter(timeout);\n var retObj = func();\n EMSglobal.criticalExit();\n return retObj\n}\n\n\n//==================================================================\n// Perform func only on thread 0\nfunction EMSmaster(func) {\n if (this.myID === 0) {\n return func();\n }\n}\n\n\n//==================================================================\n// Perform the function func once by the first thread to reach\n// the function. The final barrier is required because a\n// thread may try to execute the next single-execution region\n// before other threads have finished this region, which the EMS\n// runtime cannot tell apart. Barriers are phased, so a barrier\n// is used to prevent any thread from entering the next single-\n// execution region before this one is complete\nfunction EMSsingle(func) {\n var retObj;\n if (this.singleTask()) {\n retObj = func();\n }\n EMSbarrier();\n return retObj\n}\n\n\n//==================================================================\n// Wrapper around the EMS global barrier\nfunction EMSbarrier(timeout) {\n if (EMSglobal.inParallelContext) {\n if(typeof timeout === \"undefined\") {\n timeout = 500000; // TODO: Magic number -- long enough for errors, not load imbalance\n }\n var remaining_time = EMS.barrier.call(EMSglobal, timeout);\n if (remaining_time < 0) {\n console.log(\"EMSbarrier: ERROR -- Barrier timed out after\", timeout, \"iterations.\");\n // TODO: Probably should throw an error\n }\n return remaining_time;\n }\n return timeout;\n}\n\n\n//==================================================================\n// Utility functions for determining types\nfunction EMSisArray(a) {\n return typeof a.pop !== \"undefined\"\n}\n\nfunction EMSisObject(o) {\n return typeof o === \"object\" && !EMSisArray(o)\n}\n\nfunction EMSisDefined(x) {\n return typeof x !== \"undefined\"\n}\n\n\n//==================================================================\n// Release all resources associated with an EMS memory region\nfunction EMSdestroy(unlink_file) {\n EMSbarrier();\n if (EMSglobal.myID == 0) {\n this.data.destroy(unlink_file);\n }\n EMSbarrier();\n}\n\n\n//==================================================================\n// Creating a new EMS memory region\nfunction EMSnew(arg0, // Maximum number of elements the EMS region can hold\n heapSize, // #bytes of memory reserved for strings/arrays/objs/maps/etc\n filename // Optional filename for persistent EMS memory\n) {\n var fillIsJSON = false;\n var emsDescriptor = { // Internal EMS descriptor\n nElements: 1, // Required: Maximum number of elements in array\n heapSize: 0, // Optional, default=0: Space, in bytes, for strings, maps, objects, etc.\n mlock: 0, // Optional, 0-100% of EMS memory into RAM\n useMap: false, // Optional, default=false: Use a map from keys to indexes\n useExisting: false, // Optional, default=false: Preserve data if a file already exists\n ES6proxies: false, // Optional, default=false: Inferred EMS read/write syntax\n persist: true, // Optional, default=true: Preserve the file after threads exit\n doDataFill: false, // Optional, default=false: Data values should be initialized\n dataFill: undefined,//Optional, default=false: Value to initialize data to\n doSetFEtags: false, // Optional, initialize full/empty tags\n setFEtagsFull: true, // Optional, used only if doSetFEtags is true\n dimStride: [] // Stride factors for each dimension of multidimensional arrays\n };\n\n if (!EMSisDefined(arg0)) { // Nothing passed in, assume length 1\n emsDescriptor.dimensions = [1];\n } else {\n if (EMSisObject(arg0)) { // User passed in emsArrayDescriptor\n if (typeof arg0.dimensions !== \"undefined\") {\n if (typeof arg0.dimensions !== \"object\") {\n emsDescriptor.dimensions = [ arg0.dimensions ]\n } else {\n emsDescriptor.dimensions = arg0.dimensions\n }\n }\n if (typeof arg0.ES6proxies !== \"undefined\") {\n emsDescriptor.ES6proxies = arg0.ES6proxies;\n }\n if (typeof arg0.heapSize !== \"undefined\") {\n emsDescriptor.heapSize = arg0.heapSize;\n }\n if (typeof arg0.mlock !== \"undefined\") {\n emsDescriptor.mlock = arg0.mlock\n }\n if (typeof arg0.useMap !== \"undefined\") {\n emsDescriptor.useMap = arg0.useMap\n }\n if (typeof arg0.filename !== \"undefined\") {\n emsDescriptor.filename = arg0.filename\n }\n if (typeof arg0.persist !== \"undefined\") {\n emsDescriptor.persist = arg0.persist\n }\n if (typeof arg0.useExisting !== \"undefined\") {\n emsDescriptor.useExisting = arg0.useExisting\n }\n if(arg0.doDataFill) {\n emsDescriptor.doDataFill = arg0.doDataFill;\n if (typeof arg0.dataFill === \"object\") {\n emsDescriptor.dataFill = JSON.stringify(arg0.dataFill);\n fillIsJSON = true;\n } else {\n emsDescriptor.dataFill = arg0.dataFill;\n }\n }\n if (typeof arg0.doSetFEtags !== \"undefined\") {\n emsDescriptor.doSetFEtags = arg0.doSetFEtags\n }\n if (typeof arg0.setFEtags !== \"undefined\") {\n if (arg0.setFEtags === \"full\") {\n emsDescriptor.setFEtagsFull = true;\n } else {\n emsDescriptor.setFEtagsFull = false;\n }\n }\n if (typeof arg0.hashFunc !== \"undefined\") {\n emsDescriptor.hashFunc = arg0.hashFunc\n }\n } else {\n if (EMSisArray(arg0)) { // User passed in multi-dimensional array\n emsDescriptor.dimensions = arg0\n } else {\n if (typeof arg0 === \"number\") { // User passed in scalar 1-D array length\n emsDescriptor.dimensions = [arg0]\n } else {\n console.log(\"EMSnew: Couldn't determine type of arg0\", arg0, typeof arg0)\n }\n }\n }\n if (typeof heapSize === \"number\") {\n emsDescriptor.heapSize = heapSize;\n if (heapSize <= 0 && emsDescriptor.useMap) {\n console.log(\"Warning: New EMS array with no heap, disabling mapped keys\");\n emsDescriptor.useMap = false;\n }\n }\n if (typeof filename === \"string\") {\n emsDescriptor.filename = filename\n }\n }\n\n // Compute the stride factors for each dimension of a multidimensional array\n for (var dimN = 0; dimN < emsDescriptor.dimensions.length; dimN++) {\n emsDescriptor.dimStride.push(emsDescriptor.nElements);\n emsDescriptor.nElements *= emsDescriptor.dimensions[dimN];\n }\n if (typeof emsDescriptor.dimensions === \"undefined\") {\n emsDescriptor.dimensions = [emsDescriptor.nElements];\n }\n\n // Name the region if a name wasn't given\n if (!EMSisDefined(emsDescriptor.filename)) {\n emsDescriptor.filename = \"/EMS_region_\" + this.newRegionN;\n emsDescriptor.persist = false;\n }\n\n if (emsDescriptor.useExisting) {\n try { fs.openSync(emsDescriptor.filename, \"r\"); }\n catch (err) {\n return;\n }\n }\n\n // init() is first called from thread 0 to perform one-thread\n // only operations (ie: unlinking an old file, opening a new\n // file). After thread 0 has completed initialization, other\n // threads can safely share the EMS array.\n if (!emsDescriptor.useExisting && this.myID !== 0) EMSbarrier();\n emsDescriptor.data = this.init(emsDescriptor.nElements, emsDescriptor.heapSize, // 0, 1\n emsDescriptor.useMap, emsDescriptor.filename, // 2, 3\n emsDescriptor.persist, emsDescriptor.useExisting, // 4, 5\n emsDescriptor.doDataFill, fillIsJSON, // 6, 7\n emsDescriptor.dataFill, // 8\n emsDescriptor.doSetFEtags, // 9\n emsDescriptor.setFEtagsFull, // 10\n this.myID, this.pinThreads, this.nThreads, // 11, 12, 13\n emsDescriptor.mlock); // 14\n\n if (!emsDescriptor.useExisting && this.myID === 0) EMSbarrier();\n\n emsDescriptor.regionN = this.newRegionN;\n emsDescriptor.push = EMSpush;\n emsDescriptor.pop = EMSpop;\n emsDescriptor.enqueue = EMSenqueue;\n emsDescriptor.dequeue = EMSdequeue;\n emsDescriptor.setTag = EMSsetTag;\n emsDescriptor.write = EMSwrite;\n emsDescriptor.writeEF = EMSwriteEF;\n emsDescriptor.writeXF = EMSwriteXF;\n emsDescriptor.writeXE = EMSwriteXE;\n emsDescriptor.read = EMSread;\n emsDescriptor.readRW = EMSreadRW;\n emsDescriptor.releaseRW = EMSreleaseRW;\n emsDescriptor.readFE = EMSreadFE;\n emsDescriptor.readFF = EMSreadFF;\n emsDescriptor.faa = EMSfaa;\n emsDescriptor.cas = EMScas;\n emsDescriptor.sync = EMSsync;\n emsDescriptor.index2key = EMSindex2key;\n emsDescriptor.destroy = EMSdestroy;\n this.newRegionN++;\n EMSbarrier();\n\n // Wrap the object with a proxy\n if(emsDescriptor.ES6proxies) {\n // Setter/Getter methods for the proxy object\n // If the target is built into EMS use the built-in object,\n // otherwise read/write the EMS value without honoring the Full/Empty tag\n var EMSproxyHandler = {\n get: function(target, name) {\n if (name in target) {\n return target[name];\n } else {\n return target.read(name);\n }\n },\n set: function(target, name, value) {\n target.write(name, value);\n return true\n }\n };\n try {\n emsDescriptor = new Proxy(emsDescriptor, EMSproxyHandler);\n }\n catch (err) {\n // console.log(\"Harmony proxies not supported:\", err);\n }\n }\n\n return emsDescriptor;\n}\n\n\n//==================================================================\n// EMS object initialization, invoked by the require statement\n//\nfunction ems_wrapper(nThreadsArg, pinThreadsArg, threadingType, filename) {\n var retObj = {tasks: []};\n\n // TODO: Determining the thread ID should be done via shared memory\n if (process.env.EMS_Subtask !== undefined ) {\n retObj.myID = parseInt(process.env.EMS_Subtask);\n } else {\n retObj.myID = 0;\n }\n\n var pinThreads = false;\n if (typeof pinThreadsArg === \"boolean\") {\n pinThreads = pinThreadsArg;\n }\n\n var nThreads;\n nThreads = parseInt(nThreadsArg);\n if (!(nThreads > 0)) {\n if (process.env.EMS_Ntasks !== undefined) {\n nThreads = parseInt(process.env.EMS_Ntasks);\n\t\t\tif (!(nThreads > 0)) {\n\t\t\t\tconsole.log(\"EMS: Number of nodes must be > 0. Input:\", nThreadsArg, \"EnvVar:\", nThreads);\n\t\t\t\tprocess.exit(1);\n\t\t\t}\n } else {\n console.log(\"EMS: Must declare number of nodes to use. Input:\" + nThreadsArg);\n process.exit(1);\n }\n }\n\n var domainName = \"/EMS_MainDomain\";\n if (filename) domainName = filename;\n // All arguments are defined -- now do the EMS initialization\n retObj.data = EMS.initialize(0, 0, // 0= # elements, 1=Heap Size\n false, // 2 = useMap\n domainName, false, false, // 3=name, 4=persist, 5=useExisting\n false, false, undefined, // 6=doDataFill, 7=fillIsJSON, 8=fillValue\n false, false, retObj.myID, // 9=doSetFEtags, 10=setFEtags, 11=EMS myID\n pinThreads, nThreads, 99); // 12=pinThread, 13=nThreads, 14=pctMlock\n\n var targetScript;\n switch (threadingType) {\n case undefined:\n case \"bsp\":\n targetScript = process.argv[1];\n threadingType = \"bsp\";\n retObj.inParallelContext = true;\n break;\n case \"fj\":\n targetScript = \"./EMSthreadStub\";\n retObj.inParallelContext = false;\n break;\n case \"user\":\n targetScript = undefined;\n retObj.inParallelContext = false;\n break;\n default:\n console.log(\"EMS: Unknown threading model type:\", threadingType);\n retObj.inParallelContext = false;\n break;\n }\n\n // The master thread has completed initialization, other threads may now\n // safely execute.\n if (targetScript !== undefined && retObj.myID === 0) {\n var emsThreadStub =\n \"// Automatically Generated EMS Slave Thread Script\\n\" +\n \"// To edit this file, see ems.js:emsThreadStub()\\n\" +\n \"var ems = require(\\\"ems\\\")(parseInt(process.env.EMS_Ntasks));\\n\" +\n \"process.on(\\\"message\\\", function(msg) {\\n\" +\n \" eval(\\\"func = \\\" + msg.func);\\n\" +\n \" func.apply(null, msg.args);\\n\" +\n \"} );\\n\";\n fs.writeFileSync('./EMSthreadStub.js', emsThreadStub, {flag: 'w+'});\n process.env.EMS_Ntasks = nThreads;\n for (var taskN = 1; taskN < nThreads; taskN++) {\n process.env.EMS_Subtask = taskN;\n retObj.tasks.push(\n child_process.fork(targetScript,\n process.argv.slice(2, process.argv.length)));\n }\n }\n\n retObj.nThreads = nThreads;\n retObj.threadingType = threadingType;\n retObj.pinThreads = pinThreads;\n retObj.domainName = domainName;\n retObj.newRegionN = 0;\n retObj.init = EMS.initialize;\n retObj.new = EMSnew;\n retObj.critical = EMScritical;\n retObj.criticalEnter = EMS.criticalEnter;\n retObj.criticalExit = EMS.criticalExit;\n retObj.master = EMSmaster;\n retObj.single = EMSsingle;\n retObj.diag = EMSdiag;\n retObj.parallel = EMSparallel;\n retObj.barrier = EMSbarrier;\n retObj.parForEach = EMSparForEach;\n retObj.tmStart = EMStmStart;\n retObj.tmEnd = EMStmEnd;\n retObj.loopInit = EMS.loopInit;\n retObj.loopChunk = EMS.loopChunk;\n EMSglobal = retObj;\n EMSglobal.mmapID = retObj.data.mmapID;\n return retObj;\n}\n\nems_wrapper.initialize = ems_wrapper;\nmodule.exports = ems_wrapper;\n"
},
{
"path": "nodejs/nodejs.cc",
"content": "/*-----------------------------------------------------------------------------+\n | Extended Memory Semantics (EMS) Version 1.6.1 |\n | http://mogill.com/ jace@mogill.com |\n +-----------------------------------------------------------------------------+\n | Copyright (c) 2011-2014, Synthetic Semantics LLC. All rights reserved. |\n | Copyright (c) 2019 Aleksander J Budzynowski. Update NAN to N-API. |\n | Copyright (c) 2015-2020, Jace A Mogill. All rights reserved. |\n | |\n | Redistribution and use in source and binary forms, with or without |\n | modification, are permitted provided that the following conditions are met: |\n | * Redistributions of source code must retain the above copyright |\n | notice, this list of conditions and the following disclaimer. |\n | * Redistributions in binary form must reproduce the above copyright |\n | notice, this list of conditions and the following disclaimer in the |\n | documentation and/or other materials provided with the distribution. |\n | * Neither the name of the Synthetic Semantics nor the names of its |\n | contributors may be used to endorse or promote products derived |\n | from this software without specific prior written permission. |\n | |\n | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS |\n | \"AS IS\" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT |\n | LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR |\n | A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL SYNTHETIC |\n | SEMANTICS LLC BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, |\n | EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, |\n | PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR |\n | PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF |\n | LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING |\n | NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS |\n | SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. |\n | |\n +-----------------------------------------------------------------------------*/\n#include \n\n#include \"nodejs.h\"\n#include \"../src/ems.h\"\n#include \"../src/ems_types.h\"\n\n/**\n * Convert a NAPI object to an EMS object stored on the stack\n * @param napiValue Source Napi object\n * @param emsValue Target EMS object\n * @param stringIsJSON You know who\n * @return True if successful converting\n */\n#define NAPI_OBJ_2_EMS_OBJ(napiValue, emsValue, stringIsJSON) { \\\n emsValue.type = NapiObjToEMStype(napiValue, stringIsJSON); \\\n switch (emsValue.type) { \\\n case EMS_TYPE_BOOLEAN: { \\\n bool tmp = napiValue.As(); \\\n emsValue.value = (void *) tmp; \\\n break; \\\n } \\\n case EMS_TYPE_INTEGER: { \\\n int64_t tmp = napiValue.As(); \\\n emsValue.value = (void *) tmp; \\\n } \\\n break; \\\n case EMS_TYPE_FLOAT: { \\\n EMSulong_double alias = {.d = napiValue.As()};\\\n emsValue.value = (void *) alias.u64; \\\n } \\\n break; \\\n case EMS_TYPE_JSON: \\\n case EMS_TYPE_STRING: { \\\n std::string s = napiValue.As().Utf8Value(); \\\n emsValue.length = s.length() + 1; /* +1 for trailing NULL */\\\n struct rlimit rl; \\\n if (getrlimit(RLIMIT_STACK, &rl) == 0) { \\\n if (emsValue.length > rl.rlim_cur) { \\\n THROW_TYPE_ERROR(QUOTE(__FUNCTION__) \" ERROR: Unable to allocate scratch memory for serialized value. Exceeds current stack size limit.\"); \\\n } \\\n } else { \\\n THROW_ERROR(\" ERROR: Unable to get stack limit.\"); \\\n } \\\n emsValue.value = alloca(emsValue.length); \\\n memcpy(emsValue.value, s.c_str(), emsValue.length); \\\n } \\\n break; \\\n case EMS_TYPE_UNDEFINED: \\\n emsValue.value = (void *) 0xbeeff00d; \\\n break; \\\n default: \\\n THROW_TYPE_ERROR(QUOTE(__FUNCTION__) \" ERROR: Invalid value type\");\\\n } \\\n }\n\n\n/**\n * Convert an EMS object into a Napi object\n * @param env Napi Env object\n * @param emsValue Source EMS object\n * @return converted value\n */\nstatic Napi::Value inline\nems2napiReturnValue(Napi::Env env, EMSvalueType *emsValue) {\n switch(emsValue->type) {\n case EMS_TYPE_BOOLEAN: {\n return Napi::Value::From(env, (bool) emsValue->value);\n }\n break;\n case EMS_TYPE_INTEGER: {\n int32_t retInt = ((int64_t) emsValue->value) & 0xffffffff; /* TODO: Bug -- only 32 bits of 64? */\n return Napi::Number::New(env, retInt);\n }\n break;\n case EMS_TYPE_FLOAT: {\n EMSulong_double alias = {.u64 = (uint64_t) emsValue->value};\n return Napi::Number::New(env, alias.d);\n }\n break;\n case EMS_TYPE_JSON: {\n Napi::Object retObj = Napi::Object::New(env);\n retObj.Set(\"data\", Napi::String::New(env, (char *) emsValue->value));\n return retObj;\n }\n break;\n case EMS_TYPE_STRING: {\n return Napi::String::New(env, (char *) emsValue->value);\n }\n break;\n case EMS_TYPE_UNDEFINED: {\n return env.Undefined();\n }\n break;\n default:\n THROW_TYPE_ERROR(\"ems2napiReturnValue - ERROR: Invalid type of data read from memory\");\n }\n}\n\n\nNapi::Value NodeJScriticalEnter(const Napi::CallbackInfo& info) {\n Napi::Env env = info.Env();\n NODE_MMAPID_DECL;\n int64_t timeout;\n if (info.Length() == 1) {\n timeout = info[0].As();\n } else {\n THROW_ERROR(\"NodeJScriticalEner: invalid or missing timeout duration\");\n }\n int timeRemaining = EMScriticalEnter(mmapID, (int) timeout);\n if (timeRemaining <= 0) {\n THROW_ERROR(\"NodeJScriticalEnter: Unable to enter critical region before timeout\");\n } else {\n return Napi::Value::From(env, timeRemaining);\n }\n}\n\n\nNapi::Value NodeJScriticalExit(const Napi::CallbackInfo& info) {\n Napi::Env env = info.Env();\n NODE_MMAPID_DECL;\n bool success = EMScriticalExit(mmapID);\n if (!success) {\n THROW_ERROR(\"NodeJScriticalExit: critical region mutex lost while locked?!\");\n } else {\n return Napi::Value::From(env, success);\n }\n}\n\n\nNapi::Value NodeJSbarrier(const Napi::CallbackInfo& info) {\n Napi::Env env = info.Env();\n NODE_MMAPID_DECL;\n int timeout;\n if (info.Length() == 1) {\n timeout = info[0].As();\n } else {\n THROW_ERROR(\"NodeJSbarrier: invalid or missing timeout duration\");\n }\n int timeRemaining = EMSbarrier(mmapID, timeout);\n if (timeRemaining <= 0) {\n THROW_ERROR(\"NodeJSbarrer: Failed to sync at barrier\");\n } else {\n return Napi::Value::From(env, timeRemaining);\n }\n}\n\n\nNapi::Value NodeJSsingleTask(const Napi::CallbackInfo& info) {\n Napi::Env env = info.Env();\n NODE_MMAPID_DECL;\n bool did_work = EMSsingleTask(mmapID);\n return Napi::Boolean::New(env, did_work);\n}\n\n\nNapi::Value NodeJScas(const Napi::CallbackInfo& info) {\n Napi::Env env = info.Env();\n EMSvalueType returnValue = EMS_VALUE_TYPE_INITIALIZER;\n STACK_ALLOC_AND_CHECK_KEY_ARG;\n EMSvalueType oldVal = EMS_VALUE_TYPE_INITIALIZER;\n EMSvalueType newVal = EMS_VALUE_TYPE_INITIALIZER;\n if (info.Length() != 3) {\n THROW_ERROR(SOURCE_LOCATION \": Called with wrong number of args.\");\n }\n NAPI_OBJ_2_EMS_OBJ(info[1], oldVal, false);\n NAPI_OBJ_2_EMS_OBJ(info[2], newVal, false);\n if (!EMScas(mmapID, &key, &oldVal, &newVal, &returnValue)) {\n THROW_ERROR(\"NodeJScas: Failed to get a valid old value\");\n }\n return ems2napiReturnValue(env, &returnValue);\n}\n\n\nNapi::Value NodeJSfaa(const Napi::CallbackInfo& info) {\n Napi::Env env = info.Env();\n EMSvalueType returnValue = EMS_VALUE_TYPE_INITIALIZER;\n STACK_ALLOC_AND_CHECK_KEY_ARG;\n STACK_ALLOC_AND_CHECK_VALUE_ARG(1);\n bool success = EMSfaa(mmapID, &key, &value, &returnValue);\n if (!success) {\n THROW_ERROR(\"NodeJSfaa: Failed to get a valid old value\");\n }\n return ems2napiReturnValue(env, &returnValue);\n}\n\n\nNapi::Value NodeJSpush(const Napi::CallbackInfo& info) {\n Napi::Env env = info.Env();\n NODE_MMAPID_DECL;\n STACK_ALLOC_AND_CHECK_VALUE_ARG(0);\n int returnValue = EMSpush(mmapID, &value);\n return Napi::Value::From(env, returnValue);\n}\n\n\nNapi::Value NodeJSpop(const Napi::CallbackInfo& info) {\n Napi::Env env = info.Env();\n EMSvalueType returnValue = EMS_VALUE_TYPE_INITIALIZER;\n NODE_MMAPID_DECL;\n bool success = EMSpop(mmapID, &returnValue);\n if (!success) {\n THROW_ERROR(\"NodeJSpop: Failed to pop a value off the stack\");\n }\n return ems2napiReturnValue(env, &returnValue);\n}\n\n\nNapi::Value NodeJSenqueue(const Napi::CallbackInfo& info) {\n Napi::Env env = info.Env();\n NODE_MMAPID_DECL;\n STACK_ALLOC_AND_CHECK_VALUE_ARG(0);\n int returnValue = EMSenqueue(mmapID, &value);\n return Napi::Value::From(env, returnValue);\n}\n\n\nNapi::Value NodeJSdequeue(const Napi::CallbackInfo& info) {\n Napi::Env env = info.Env();\n EMSvalueType returnValue = EMS_VALUE_TYPE_INITIALIZER;\n NODE_MMAPID_DECL;\n bool success = EMSdequeue(mmapID, &returnValue);\n if (!success) {\n THROW_ERROR(\"NodeJSdequeue: Failed to dequeue a value\");\n }\n return ems2napiReturnValue(env, &returnValue);\n}\n\n\nNapi::Value NodeJSloopInit(const Napi::CallbackInfo& info) {\n Napi::Env env = info.Env();\n NODE_MMAPID_DECL;\n if (info.Length() != 4) {\n THROW_ERROR(\"NodeJSloopInit: Wrong number of args\");\n }\n\n int32_t start = (int32_t) info[0].As();\n int32_t end = (int32_t) info[1].As();\n int schedule_mode;\n std::string sched_string = info[2].As().Utf8Value();\n if (sched_string.compare(\"guided\") == 0) {\n schedule_mode = EMS_SCHED_GUIDED;\n } else {\n if (sched_string.compare(\"dynamic\") == 0) {\n schedule_mode = EMS_SCHED_DYNAMIC;\n } else {\n THROW_ERROR(\"NodeJSloopInit: Unknown/invalid schedule mode\");\n }\n }\n int32_t minChunk = (int32_t) info[3].As();\n\n bool success = EMSloopInit(mmapID, start, end, minChunk, schedule_mode);\n if (!success) {\n THROW_ERROR(\"NodeJSloopInit: Unknown failure to initalize loop\");\n } else {\n return Napi::Value::From(env, success);\n }\n}\n\n\nNapi::Value NodeJSloopChunk(const Napi::CallbackInfo& info) {\n Napi::Env env = info.Env();\n NODE_MMAPID_DECL;\n if (info.Length() != 0) {\n THROW_ERROR(\"NodeJSloopChunk: Arguments provided, but none accepted\");\n }\n int32_t start, end;\n EMSloopChunk(mmapID, &start, &end); // Unusued return value\n\n Napi::Object retObj = Napi::Object::New(env);\n retObj.Set(\"start\", Napi::Value::From(env, start));\n retObj.Set(\"end\", Napi::Value::From(env, end));\n return retObj;\n}\n\n\n//--------------------------------------------------------------\nNapi::Value NodeJSread(const Napi::CallbackInfo& info) {\n Napi::Env env = info.Env();\n EMSvalueType returnValue = EMS_VALUE_TYPE_INITIALIZER;\n STACK_ALLOC_AND_CHECK_KEY_ARG;\n if (!EMSread(mmapID, &key, &returnValue)) {\n THROW_ERROR(QUOTE(__FUNCTION__) \": Unable to read (no return value) from EMS.\");\n } else {\n return ems2napiReturnValue(env, &returnValue);\n }\n}\n\n\nNapi::Value NodeJSreadFE(const Napi::CallbackInfo& info) {\n Napi::Env env = info.Env();\n EMSvalueType returnValue = EMS_VALUE_TYPE_INITIALIZER;\n STACK_ALLOC_AND_CHECK_KEY_ARG;\n if (!EMSreadFE(mmapID, &key, &returnValue)) {\n THROW_ERROR(QUOTE(__FUNCTION__) \": Unable to read (no return value) from EMS.\");\n } else {\n return ems2napiReturnValue(env, &returnValue);\n }\n}\n\n\nNapi::Value NodeJSreadFF(const Napi::CallbackInfo& info) {\n Napi::Env env = info.Env();\n EMSvalueType returnValue = EMS_VALUE_TYPE_INITIALIZER;\n STACK_ALLOC_AND_CHECK_KEY_ARG;\n if (!EMSreadFF(mmapID, &key, &returnValue)) {\n THROW_ERROR(QUOTE(__FUNCTION__) \": Unable to read (no return value) from EMS.\");\n } else {\n return ems2napiReturnValue(env, &returnValue);\n }\n}\n\n\nNapi::Value NodeJSreadRW(const Napi::CallbackInfo& info) {\n Napi::Env env = info.Env();\n EMSvalueType returnValue = EMS_VALUE_TYPE_INITIALIZER;\n STACK_ALLOC_AND_CHECK_KEY_ARG;\n if (!EMSreadRW(mmapID, &key, &returnValue)) {\n THROW_ERROR(QUOTE(__FUNCTION__) \": Unable to read (no return value) from EMS.\");\n } else {\n return ems2napiReturnValue(env, &returnValue);\n }\n}\n\n\nNapi::Value NodeJSreleaseRW(const Napi::CallbackInfo& info) {\n Napi::Env env = info.Env();\n STACK_ALLOC_AND_CHECK_KEY_ARG;\n int nReadersActive = EMSreleaseRW(mmapID, &key);\n if (nReadersActive < 0) {\n THROW_ERROR(\"NodeJSreleaseRW: Invalid index for key, or index key in bad state\");\n } else {\n return Napi::Value::From(env, nReadersActive);\n }\n}\n\n// ====================================================\n\nNapi::Value NodeJSwrite(const Napi::CallbackInfo& info) {\n Napi::Env env = info.Env();\n STACK_ALLOC_AND_CHECK_KEY_ARG;\n STACK_ALLOC_AND_CHECK_VALUE_ARG(1);\n bool returnValue = EMSwrite(mmapID, &key, &value);\n return Napi::Value::From(env, returnValue);\n}\n\n\nNapi::Value NodeJSwriteEF(const Napi::CallbackInfo& info) {\n Napi::Env env = info.Env();\n STACK_ALLOC_AND_CHECK_KEY_ARG;\n STACK_ALLOC_AND_CHECK_VALUE_ARG(1);\n bool returnValue = EMSwriteEF(mmapID, &key, &value);\n return Napi::Value::From(env, returnValue);\n}\n\n\nNapi::Value NodeJSwriteXF(const Napi::CallbackInfo& info) {\n Napi::Env env = info.Env();\n STACK_ALLOC_AND_CHECK_KEY_ARG;\n STACK_ALLOC_AND_CHECK_VALUE_ARG(1);\n bool returnValue = EMSwriteXF(mmapID, &key, &value);\n return Napi::Value::From(env, returnValue);\n}\n\n\nNapi::Value NodeJSwriteXE(const Napi::CallbackInfo& info) {\n Napi::Env env = info.Env();\n STACK_ALLOC_AND_CHECK_KEY_ARG;\n STACK_ALLOC_AND_CHECK_VALUE_ARG(1);\n bool returnValue = EMSwriteXE(mmapID, &key, &value);\n return Napi::Value::From(env, returnValue);\n}\n\n\nNapi::Value NodeJSsetTag(const Napi::CallbackInfo& info) {\n Napi::Env env = info.Env();\n STACK_ALLOC_AND_CHECK_KEY_ARG;\n STACK_ALLOC_AND_CHECK_VALUE_ARG(1); // Bool -- is full\n bool success = EMSsetTag(mmapID, &key, (bool)value.value);\n if (success) {\n return Napi::Value::From(env, true);\n } else {\n THROW_ERROR(\"NodeJSsetTag: Invalid key, unable to set tag\");\n }\n}\n\n\nNapi::Value NodeJSsync(const Napi::CallbackInfo& info) {\n Napi::Env env = info.Env();\n NODE_MMAPID_DECL;\n fprintf(stderr, \"NodeJSsync: WARNING: sync is not implemented\\n\");\n bool success = EMSsync(mmapID);\n return Napi::Value::From(env, success);\n}\n\n\nNapi::Value NodeJSindex2key(const Napi::CallbackInfo& info) {\n Napi::Env env = info.Env();\n NODE_MMAPID_DECL;\n EMSvalueType key = EMS_VALUE_TYPE_INITIALIZER;\n int idx = (size_t) (int64_t) info[0].As();\n if ( !EMSindex2key(mmapID, idx, &key) ) {\n fprintf(stderr, \"NodeJSindex2key: Error converting index (%d) to key\\n\", idx);\n }\n return ems2napiReturnValue(env, &key);\n}\n\n\nNapi::Value NodeJSdestroy(const Napi::CallbackInfo& info) {\n Napi::Env env = info.Env();\n NODE_MMAPID_DECL;\n bool do_unlink = info[0].As();\n bool success = EMSdestroy(mmapID, do_unlink);\n if (success) {\n return Napi::Value::From(env, true);\n } else {\n THROW_ERROR(\"NodeJSdestroy: Failed to destroy EMS array\");\n }\n};\n\n\n//==================================================================\n// EMS Entry Point: Allocate and initialize the EMS domain memory\nNapi::Value NodeJSinitialize(const Napi::CallbackInfo& info) {\n Napi::Env env = info.Env();\n if (info.Length() != 15) {\n THROW_ERROR(\"NodeJSinitialize: Incorrect number of arguments\");\n }\n EMSvalueType fillData = EMS_VALUE_TYPE_INITIALIZER;\n\n // Parse all the arguments\n int64_t nElements = info[0].As();\n size_t heapSize = (int64_t) info[1].As();\n bool useMap = info[2].As();\n std::string filestr(info[3].As().Utf8Value());\n const char *filename = filestr.c_str();\n bool persist = info[4].As();\n bool useExisting = info[5].As();\n bool doDataFill = info[6].As();\n bool fillIsJSON = info[7].As();\n // 8 = Data Fill type TBD during fill\n bool doSetFEtags = info[9].As();\n bool setFEtags = info[10].As();\n EMSmyID = (int) info[11].As();\n bool pinThreads = info[12].As();\n int32_t nThreads = info[13].As();\n int32_t pctMLock = info[14].As();\n\n if (doDataFill) {\n NAPI_OBJ_2_EMS_OBJ(info[8], fillData, fillIsJSON);\n if (doDataFill && (fillData.type == EMS_TYPE_JSON || fillData.type == EMS_TYPE_STRING)) {\n // Copy the default values to the heap because napiObj2EMSval copies them to the stack\n void *valueOnStack = fillData.value;\n fillData.value = malloc(fillData.length + 1);\n if (fillData.value == NULL) {\n THROW_ERROR(\"NodeJSinitialize: failed to allocate the default value's storage on the heap\");\n }\n memcpy(fillData.value, valueOnStack, fillData.length + 1);\n }\n }\n\n int emsBufN = EMSinitialize(nElements, // 0\n heapSize, // 1\n useMap, // 2\n filename, // 3\n persist, // 4\n useExisting, // 5\n doDataFill, // 6 Data Fill type TBD during fill\n fillIsJSON, // 7\n &fillData, // 8\n doSetFEtags, // 9\n setFEtags, // 10\n EMSmyID, // 11\n pinThreads, // 12\n nThreads, // 13\n pctMLock); // 14\n\n if (emsBufN < 0) {\n THROW_ERROR(\"NodeJSinitialize: failed to initialize EMS array\");\n }\n\n // ========================================================================================\n Napi::Object obj = Napi::Object::New(env);\n obj.Set(Napi::String::New(env, \"mmapID\"), Napi::Value::From(env, emsBufN));\n ADD_FUNC_TO_NAPI_OBJ(obj, \"faa\", NodeJSfaa);\n ADD_FUNC_TO_NAPI_OBJ(obj, \"cas\", NodeJScas);\n ADD_FUNC_TO_NAPI_OBJ(obj, \"read\", NodeJSread);\n ADD_FUNC_TO_NAPI_OBJ(obj, \"write\", NodeJSwrite);\n ADD_FUNC_TO_NAPI_OBJ(obj, \"readRW\", NodeJSreadRW);\n ADD_FUNC_TO_NAPI_OBJ(obj, \"releaseRW\", NodeJSreleaseRW);\n ADD_FUNC_TO_NAPI_OBJ(obj, \"readFE\", NodeJSreadFE);\n ADD_FUNC_TO_NAPI_OBJ(obj, \"readFF\", NodeJSreadFF);\n ADD_FUNC_TO_NAPI_OBJ(obj, \"setTag\", NodeJSsetTag);\n ADD_FUNC_TO_NAPI_OBJ(obj, \"writeEF\", NodeJSwriteEF);\n ADD_FUNC_TO_NAPI_OBJ(obj, \"writeXF\", NodeJSwriteXF);\n ADD_FUNC_TO_NAPI_OBJ(obj, \"writeXE\", NodeJSwriteXE);\n ADD_FUNC_TO_NAPI_OBJ(obj, \"push\", NodeJSpush);\n ADD_FUNC_TO_NAPI_OBJ(obj, \"pop\", NodeJSpop);\n ADD_FUNC_TO_NAPI_OBJ(obj, \"enqueue\", NodeJSenqueue);\n ADD_FUNC_TO_NAPI_OBJ(obj, \"dequeue\", NodeJSdequeue);\n ADD_FUNC_TO_NAPI_OBJ(obj, \"sync\", NodeJSsync);\n ADD_FUNC_TO_NAPI_OBJ(obj, \"index2key\", NodeJSindex2key);\n ADD_FUNC_TO_NAPI_OBJ(obj, \"destroy\", NodeJSdestroy);\n return obj;\n}\n\n\n//---------------------------------------------------------------\nstatic Napi::Object RegisterModule(Napi::Env env, Napi::Object exports) {\n ADD_FUNC_TO_NAPI_OBJ(exports, \"initialize\", NodeJSinitialize);\n ADD_FUNC_TO_NAPI_OBJ(exports, \"barrier\", NodeJSbarrier);\n ADD_FUNC_TO_NAPI_OBJ(exports, \"singleTask\", NodeJSsingleTask);\n ADD_FUNC_TO_NAPI_OBJ(exports, \"criticalEnter\", NodeJScriticalEnter);\n ADD_FUNC_TO_NAPI_OBJ(exports, \"criticalExit\", NodeJScriticalExit);\n ADD_FUNC_TO_NAPI_OBJ(exports, \"loopInit\", NodeJSloopInit);\n ADD_FUNC_TO_NAPI_OBJ(exports, \"loopChunk\", NodeJSloopChunk);\n return exports;\n}\n\n\nNODE_API_MODULE(ems, RegisterModule);\n"
},
{
"path": "nodejs/nodejs.h",
"content": "/*-----------------------------------------------------------------------------+\n | Extended Memory Semantics (EMS) Version 1.6.1 |\n | http://mogill.com/ jace@mogill.com |\n +-----------------------------------------------------------------------------+\n | Copyright (c) 2011-2014, Synthetic Semantics LLC. All rights reserved. |\n | Copyright (c) 2019 Aleksander J Budzynowski. Update NAN to N-API. |\n | Copyright (c) 2015-2020, Jace A Mogill. All rights reserved. |\n | |\n | Redistribution and use in source and binary forms, with or without |\n | modification, are permitted provided that the following conditions are met: |\n | * Redistributions of source code must retain the above copyright |\n | notice, this list of conditions and the following disclaimer. |\n | * Redistributions in binary form must reproduce the above copyright |\n | notice, this list of conditions and the following disclaimer in the |\n | documentation and/or other materials provided with the distribution. |\n | * Neither the name of the Synthetic Semantics nor the names of its |\n | contributors may be used to endorse or promote products derived |\n | from this software without specific prior written permission. |\n | |\n | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS |\n | \"AS IS\" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT |\n | LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR |\n | A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL SYNTHETIC |\n | SEMANTICS LLC BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, |\n | EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, |\n | PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR |\n | PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF |\n | LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING |\n | NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS |\n | SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. |\n | |\n +-----------------------------------------------------------------------------*/\n#ifndef EMSPROJ_NODEJS_H\n#define EMSPROJ_NODEJS_H\n#define __STDC_LIMIT_MACROS\n#include \n#include \"napi.h\"\n\n#define QUOTE_ARG(x) #x\n#define QUOTE(x) QUOTE_ARG(x)\n\n#define THROW_ERROR(error) { \\\n Napi::Error::New(env, error) \\\n .ThrowAsJavaScriptException(); \\\n return env.Null(); \\\n }\n\n#define THROW_TYPE_ERROR(error) { \\\n Napi::TypeError::New(env, error) \\\n .ThrowAsJavaScriptException(); \\\n return env.Null(); \\\n }\n\n\n#define ADD_FUNC_TO_NAPI_OBJ(obj, func_name, func) \\\n { \\\n Napi::Function fn = Napi::Function::New(env, func, func_name); \\\n obj.Set(Napi::Value::From(env, func_name), fn); \\\n }\n\n#define IS_INTEGER(x) ((double)(int64_t)(x) == (double)x)\n//==================================================================\n// Determine the EMS type of a Napi argument\n#define NapiObjToEMStype(arg, stringIsJSON) \\\n( \\\n arg.IsNumber() ? \\\n (IS_INTEGER(arg.As()) ? EMS_TYPE_INTEGER : \\\n EMS_TYPE_FLOAT ) : \\\n (arg.IsString() && !stringIsJSON) ? EMS_TYPE_STRING : \\\n (arg.IsString() && stringIsJSON) ? EMS_TYPE_JSON : \\\n arg.IsBoolean() ? EMS_TYPE_BOOLEAN : \\\n arg.IsUndefined() ? EMS_TYPE_UNDEFINED: \\\n EMS_TYPE_INVALID \\\n)\n\n\n#define SOURCE_LOCATION __FILE__ \":\" QUOTE(__LINE__)\n\n#define STACK_ALLOC_AND_CHECK_KEY_ARG \\\n NODE_MMAPID_DECL; \\\n EMSvalueType key = EMS_VALUE_TYPE_INITIALIZER; \\\n if (info.Length() < 1) { \\\n Napi::Error::New(env, SOURCE_LOCATION \": missing key argument.\").ThrowAsJavaScriptException(); \\\n } else { \\\n NAPI_OBJ_2_EMS_OBJ(info[0], key, false); \\\n }\n\n\n#define STACK_ALLOC_AND_CHECK_VALUE_ARG(argNum) \\\n bool stringIsJSON = false; \\\n EMSvalueType value = EMS_VALUE_TYPE_INITIALIZER; \\\n if (info.Length() == argNum + 2) { \\\n stringIsJSON = info[argNum + 1].As(); \\\n } \\\n if (info.Length() < argNum + 1) { \\\n Napi::Error::New(env, SOURCE_LOCATION \": ERROR, wrong number of arguments for value\").ThrowAsJavaScriptException(); \\\n } else { \\\n NAPI_OBJ_2_EMS_OBJ(info[argNum], value, stringIsJSON); \\\n }\n\n#define NODE_MMAPID_DECL \\\n const int mmapID = (int) info.This().As() \\\n .Get(\"mmapID\").As()\n\n\nNapi::Value NodeJScriticalEnter(const Napi::CallbackInfo& info);\nNapi::Value NodeJScriticalExit(const Napi::CallbackInfo& info);\nNapi::Value NodeJSbarrier(const Napi::CallbackInfo& info);\nNapi::Value NodeJSsingleTask(const Napi::CallbackInfo& info);\nNapi::Value NodeJScas(const Napi::CallbackInfo& info);\nNapi::Value NodeJSfaa(const Napi::CallbackInfo& info);\nNapi::Value NodeJSpush(const Napi::CallbackInfo& info);\nNapi::Value NodeJSpop(const Napi::CallbackInfo& info);\nNapi::Value NodeJSenqueue(const Napi::CallbackInfo& info);\nNapi::Value NodeJSdequeue(const Napi::CallbackInfo& info);\nNapi::Value NodeJSloopInit(const Napi::CallbackInfo& info);\nNapi::Value NodeJSloopChunk(const Napi::CallbackInfo& info);\n//--------------------------------------------------------------\nNapi::Value NodeJSread(const Napi::CallbackInfo& info);\nNapi::Value NodeJSreadRW(const Napi::CallbackInfo& info);\nNapi::Value NodeJSreleaseRW(const Napi::CallbackInfo& info);\nNapi::Value NodeJSreadFE(const Napi::CallbackInfo& info);\nNapi::Value NodeJSreadFF(const Napi::CallbackInfo& info);\nNapi::Value NodeJSwrite(const Napi::CallbackInfo& info);\nNapi::Value NodeJSwriteEF(const Napi::CallbackInfo& info);\nNapi::Value NodeJSwriteXF(const Napi::CallbackInfo& info);\nNapi::Value NodeJSwriteXE(const Napi::CallbackInfo& info);\nNapi::Value NodeJSsetTag(const Napi::CallbackInfo& info);\nNapi::Value NodeJSsync(const Napi::CallbackInfo& info);\nNapi::Value NodeJSindex2key(const Napi::CallbackInfo& info);\nNapi::Value NodeJSdestroy(const Napi::CallbackInfo& info);\n\n#endif //EMSPROJ__H\n"
},
{
"path": "package.json",
"content": "{\n \"name\": \"ems\",\n \"version\": \"1.6.1\",\n \"author\": \"Jace A Mogill \",\n \"description\": \"Persistent Shared Memory and Parallel Programming Model\",\n \"contributors\": [\n {\n \"name\": \"Jace A Mogill\",\n \"email\": \"mogill@mogill.com\"\n },\n {\n \"name\": \"Aleksander Budzynowski\",\n \"email\": \"aleks@budzynowski.com\"\n }\n ],\n \"scripts\": {\n \"test\": \"cd Tests && rm -f EMSthreadStub.js && for test in `ls *js`; do node $test 8; err=$?; echo $test \\\": ERROR=\\\" $err; if [ $err -ne 0 ] ; then exit 1; fi; done\",\n \"example\": \"cd Examples; node concurrent_Q_and_TM.js 8\"\n },\n \"main\": \"nodejs/ems.js\",\n \"repository\": {\n \"type\": \"git\",\n \"url\": \"https://github.com/mogill/ems.git\"\n },\n \"homepage\": \"https://github.com/mogill/ems.git\",\n \"keywords\": [\n \"non volatile memory\",\n \"NVM\",\n \"NVMe\",\n \"multithreading\",\n \"multithreaded\",\n \"parallel\",\n \"parallelism\",\n \"concurrency\",\n \"shared memory\",\n \"multicore\",\n \"manycore\",\n \"transactional memory\",\n \"TM\",\n \"persistent memory\",\n \"pmem\",\n \"Extended Memory Semantics\",\n \"EMS\"\n ],\n \"license\": \"BSD-3-Clause\",\n \"engines\": {\n \"node\": \">=4.0\"\n },\n \"dependencies\": {\n \"bindings\": \"^1.3.0\",\n \"node-addon-api\": \"*\"\n }\n}\n"
},
{
"path": "src/collectives.cc",
"content": "/*-----------------------------------------------------------------------------+\n | Extended Memory Semantics (EMS) Version 1.3.0 |\n | Synthetic Semantics http://www.synsem.com/ mogill@synsem.com |\n +-----------------------------------------------------------------------------+\n | Copyright (c) 2011-2014, Synthetic Semantics LLC. All rights reserved. |\n | Copyright (c) 2015-2016, Jace A Mogill. All rights reserved. |\n | |\n | Redistribution and use in source and binary forms, with or without |\n | modification, are permitted provided that the following conditions are met: |\n | * Redistributions of source code must retain the above copyright |\n | notice, this list of conditions and the following disclaimer. |\n | * Redistributions in binary form must reproduce the above copyright |\n | notice, this list of conditions and the following disclaimer in the |\n | documentation and/or other materials provided with the distribution. |\n | * Neither the name of the Synthetic Semantics nor the names of its |\n | contributors may be used to endorse or promote products derived |\n | from this software without specific prior written permission. |\n | |\n | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS |\n | \"AS IS\" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT |\n | LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR |\n | A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL SYNTHETIC |\n | SEMANTICS LLC BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, |\n | EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, |\n | PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR |\n | PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF |\n | LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING |\n | NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS |\n | SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. |\n | |\n +-----------------------------------------------------------------------------*/\n#include \"ems.h\"\n\n\n//==================================================================\n// Execute Once -- Single execution\n// OpenMP style, only first thread executes, remaining skip\nbool EMSsingleTask(int mmapID) {\n void *emsBuf = emsBufs[mmapID];\n int32_t *bufInt32 = (int32_t *) emsBuf;\n\n // Increment the tally of threads that have reached this statement\n int retval = __sync_fetch_and_add(&bufInt32[EMS_CB_SINGLE], 1);\n\n // If all threads have passed through the counter, reset fo next time\n if (retval == bufInt32[EMS_CB_NTHREADS] - 1) {\n bufInt32[EMS_CB_SINGLE] = 0;\n }\n\n // Return True if this thread was first to the counter, later\n // threads return false\n return retval == 0;\n}\n\n\n//==================================================================\n// Critical Region Entry -- 1 thread at a time passes this barrier\n//\nint EMScriticalEnter(int mmapID, int timeout) {\n RESET_NAP_TIME;\n void *emsBuf = emsBufs[mmapID];\n int32_t *bufInt32 = (int32_t *) emsBuf;\n\n // Acquire the mutual exclusion lock\n while (!__sync_bool_compare_and_swap(&(bufInt32[EMS_CB_CRITICAL]), EMS_TAG_FULL, EMS_TAG_EMPTY)\n && timeout > 0 ) {\n NANOSLEEP;\n timeout -= 1;\n }\n\n return timeout;\n}\n\n\n//==================================================================\n// Critical Region Exit\nbool EMScriticalExit(int mmapID) {\n void *emsBuf = emsBufs[mmapID];\n int32_t *bufInt32 = (int32_t *) emsBuf;\n\n // Test the mutual exclusion lock wasn't somehow lost\n if (bufInt32[EMS_CB_CRITICAL] != EMS_TAG_EMPTY) {\n return false;\n }\n\n bufInt32[EMS_CB_CRITICAL] = EMS_TAG_FULL;\n return true;\n}\n\n\n//==================================================================\n// Phase Based Global Thread Barrier\nint EMSbarrier(int mmapID, int timeout) {\n void *emsBuf = emsBufs[mmapID];\n int32_t *bufInt32 = (int32_t *) emsBuf;\n\n int barPhase = bufInt32[EMS_CB_BARPHASE]; // Determine current phase of barrier\n int retval = __sync_fetch_and_add(&bufInt32[EMS_CB_NBAR0 + barPhase], -1);\n if (retval < 0) {\n fprintf(stderr, \"EMSbarrier: Race condition at barrier\\n\");\n return false;\n }\n\n if (retval == 1) {\n // This thread was the last to reach the barrier,\n // Reset the barrier count for this phase and graduate to the next phase\n bufInt32[EMS_CB_NBAR0 + barPhase] = bufInt32[EMS_CB_NTHREADS];\n bufInt32[EMS_CB_BARPHASE] = !barPhase;\n } else {\n // Wait for the barrier phase to change, indicating the last thread arrived\n RESET_NAP_TIME;\n while (timeout > 0 && barPhase == bufInt32[EMS_CB_BARPHASE]) {\n NANOSLEEP;\n timeout -= 1;\n }\n }\n\n return timeout;\n}\n"
},
{
"path": "src/ems.cc",
"content": "/*-----------------------------------------------------------------------------+\n | Extended Memory Semantics (EMS) Version 1.6.1 |\n | http://mogill.com/ jace@mogill.com |\n +-----------------------------------------------------------------------------+\n | Copyright (c) 2011-2014, Synthetic Semantics LLC. All rights reserved. |\n | Copyright (c) 2019 Aleksander J Budzynowski. Update NAN to N-API. |\n | Copyright (c) 2015-2020, Jace A Mogill. All rights reserved. |\n | |\n | Redistribution and use in source and binary forms, with or without |\n | modification, are permitted provided that the following conditions are met: |\n | * Redistributions of source code must retain the above copyright |\n | notice, this list of conditions and the following disclaimer. |\n | * Redistributions in binary form must reproduce the above copyright |\n | notice, this list of conditions and the following disclaimer in the |\n | documentation and/or other materials provided with the distribution. |\n | * Neither the name of the Synthetic Semantics nor the names of its |\n | contributors may be used to endorse or promote products derived |\n | from this software without specific prior written permission. |\n | |\n | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS |\n | \"AS IS\" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT |\n | LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR |\n | A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL SYNTHETIC |\n | SEMANTICS LLC BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, |\n | EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, |\n | PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR |\n | PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF |\n | LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING |\n | NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS |\n | SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. |\n | |\n +-----------------------------------------------------------------------------*/\n#include \"ems.h\"\n\n//==================================================================\n// Resolve External Declarations\n//\nint EMSmyID = -1; // EMS Process ID\nchar *emsBufs[EMS_MAX_N_BUFS] = { NULL };\nsize_t emsBufLengths[EMS_MAX_N_BUFS] = { 0 };\nchar emsBufFilenames[EMS_MAX_N_BUFS][MAX_FNAME_LEN] = { { 0 } };\n\n//==================================================================\n// Wrappers around memory allocator to ensure mutual exclusion\n// The buddy memory allocator is not thread safe, so this is necessary for now.\n// It can be used as the hook to wrap the non-threaded allocator with a\n// multiplexor of several independent regions.\n//\n// Returns the byte offset in the EMS data space of the space allocated\n//\nsize_t emsMutexMem_alloc(struct emsMem *heap, // Base of EMS malloc structs\n size_t len, // Number of bytes to allocate\n volatile char *mutex) // Pointer to the mem allocator's mutex\n{\n RESET_NAP_TIME;\n // Wait until we acquire the allocator's mutex\n while (!__sync_bool_compare_and_swap(mutex, EMS_TAG_EMPTY, EMS_TAG_FULL)) {\n NANOSLEEP;\n }\n size_t retval = emsMem_alloc(heap, len);\n *mutex = EMS_TAG_EMPTY; // Unlock the allocator's mutex\n return (retval);\n}\n\n\nvoid emsMutexMem_free(struct emsMem *heap, // Base of EMS malloc structs\n size_t addr, // Offset of alloc'd block in EMS memory\n volatile char *mutex) // Pointer to the mem allocator's mutex\n{\n RESET_NAP_TIME;\n // Wait until we acquire the allocator's mutex\n while (!__sync_bool_compare_and_swap(mutex, EMS_TAG_EMPTY, EMS_TAG_FULL)) {\n NANOSLEEP;\n }\n emsMem_free(heap, addr);\n *mutex = EMS_TAG_EMPTY; // Unlock the allocator's mutex\n}\n\n\n\n\n\n//==================================================================\n// Convert any type of key to an index\n//\nint64_t EMSkey2index(void *emsBuf, EMSvalueType *key, bool is_mapped) {\n volatile EMStag_t *bufTags = (EMStag_t *) emsBuf;\n volatile int64_t *bufInt64 = (int64_t *) emsBuf;\n volatile double *bufDouble = (double *) emsBuf;\n const char *bufChar = (char *) emsBuf;\n\n int64_t idx = 0;\n switch (key->type) {\n case EMS_TYPE_BOOLEAN:\n if ((bool) key->value) idx = 1;\n else idx = 0;\n break;\n case EMS_TYPE_INTEGER:\n idx = llabs((int64_t) key->value);\n break;\n case EMS_TYPE_FLOAT:\n idx = llabs((int64_t) key->value);\n break;\n case EMS_TYPE_STRING:\n idx = EMShashString((char *) key->value);\n break;\n case EMS_TYPE_UNDEFINED:\n fprintf(stderr, \"EMS ERROR: EMSkey2index keyType is defined as Undefined\\n\");\n return -1;\n default:\n fprintf(stderr, \"EMS ERROR: EMSkey2index keyType(%d) is unknown\\n\", key->type);\n return -1;\n }\n\n int nTries = 0;\n bool matched = false;\n bool notPresent = false;\n EMStag_t mapTags;\n if (is_mapped) {\n while (nTries < MAX_OPEN_HASH_STEPS && !matched && !notPresent) {\n idx = idx % bufInt64[EMScbData(EMS_ARR_NELEM)];\n // Wait until the map key is FULL, mark it busy while map lookup is performed\n mapTags.byte = EMStransitionFEtag(&bufTags[EMSmapTag(idx)], NULL, EMS_TAG_FULL, EMS_TAG_BUSY, EMS_TAG_ANY);\n if (mapTags.tags.type == key->type) {\n switch (key->type) {\n case EMS_TYPE_BOOLEAN:\n case EMS_TYPE_INTEGER:\n if ((int64_t) key->value == bufInt64[EMSmapData(idx)]) {\n matched = true;\n }\n break;\n case EMS_TYPE_FLOAT: {\n EMSulong_double alias;\n alias.u64 = (uint64_t) key->value;\n if (alias.d == bufDouble[EMSmapData(idx)]) {\n matched = true;\n }\n }\n break;\n case EMS_TYPE_STRING: {\n int64_t keyStrOffset = bufInt64[EMSmapData(idx)];\n if (strcmp((const char *) key->value, EMSheapPtr(keyStrOffset)) == 0) {\n matched = true;\n }\n }\n break;\n case EMS_TYPE_UNDEFINED:\n // Nothing hashed to this map index yet, so the key does not exist\n notPresent = true;\n break;\n default:\n fprintf(stderr, \"EMS ERROR: EMSreadIndexMap: Unknown mem type\\n\");\n matched = true;\n }\n }\n if (mapTags.tags.type == EMS_TYPE_UNDEFINED) notPresent = true;\n bufTags[EMSmapTag(idx)].tags.fe = EMS_TAG_FULL;\n if (!matched) {\n // No match, set this Map entry back to full and try again\n nTries++;\n idx++;\n }\n }\n if (!matched) { idx = -1; }\n }\n\n int64_t retval = idx % bufInt64[EMScbData(EMS_ARR_NELEM)];\n return retval;\n}\n\n\n\n#if 0\n//==================================================================\n// Callback for destruction of an EMS array\n//\nstatic void EMSarrFinalize(char *data, void *hint) {\n //fprintf(stderr, \"%d: EMSarrFinalize data=%lx hint=%lx %\" PRIu64 \"\\n\", EMSmyID, data, hint, hint);\n munmap(data, (size_t) hint);\n\n {\n v8::HandleScope scope;\n EMS_DECL(args);\n size_t length = buffer->handle_->GetIndexedPropertiesExternalArrayDataLength();\n \n if(-1 == munmap(emsBuf, length)) return v8::False();\n \n // JQM TODO shoud unlink here?\n fprintf(stderr, \"Unmap -- should also unlink\\n\");\n buffer->handle_->SetIndexedPropertiesToExternalArrayData(NULL, v8::kExternalUnsignedByteArray, 0);\n buffer->handle_.Set(length_symbol, v8::Integer::NewFromUnsigned(0));\n buffer->handle_.Dispose();\n args.This().Set(length_symbol, v8::Integer::NewFromUnsigned(0));\n \n return v8::True();\n }\n\n}\n#endif\n\n\n\n\n//==================================================================\n// Wait until the FE tag is a particular state, then transition it to the new state\n// Return new tag state\n//\nunsigned char EMStransitionFEtag(EMStag_t volatile *tag, EMStag_t volatile *mapTag,\n unsigned char oldFE, unsigned char newFE, unsigned char oldType) {\n RESET_NAP_TIME;\n EMStag_t oldTag; // Desired tag value to start of the transition\n EMStag_t newTag; // Tag value at the end of the transition\n EMStag_t volatile memTag; // Tag value actually stored in memory\n memTag.byte = tag->byte;\n while (oldType == EMS_TAG_ANY || memTag.tags.type == oldType) {\n oldTag.byte = memTag.byte; // Copy current type and RW count information\n oldTag.tags.fe = oldFE; // Set the desired start tag state\n newTag.byte = memTag.byte; // Copy current type and RW count information\n newTag.tags.fe = newFE; // Set the final tag state\n\n // Attempt to transition the state from old to new\n memTag.byte = __sync_val_compare_and_swap(&(tag->byte), oldTag.byte, newTag.byte);\n if (memTag.byte == oldTag.byte) {\n return (newTag.byte);\n } else {\n // Allow preemptive map acquisition while waiting for data\n if (mapTag) { mapTag->tags.fe = EMS_TAG_FULL; }\n NANOSLEEP;\n if (mapTag) { EMStransitionFEtag(mapTag, NULL, EMS_TAG_FULL, EMS_TAG_BUSY, EMS_TAG_ANY); }\n memTag.byte = tag->byte; // Re-load tag in case was transitioned by another thread\n }\n }\n return (memTag.byte);\n}\n\n\n//==================================================================\n// Hash a string into an integer\n//\nint64_t EMShashString(const char *key) {\n // TODO BUG MAGIC Max key length\n int charN = 0;\n uint64_t hash = 0;\n while (key[charN] != 0) {\n hash = key[charN] + (hash << 6) + (hash << 16) - hash;\n charN++;\n }\n hash *= 1191613; // Further scramble to prevent close strings from having close indexes\n return (llabs((int64_t) hash));\n}\n\n\n//==================================================================\n// Find the matching map key, if not present, find the\n// next available open address.\n// Reads map key when full and marks Busy to perform comparisons,\n// if it is not a match the data is marked full again, but if it does\n// match, the map key is left empty and this function\n// returns the index of an existing or available array element.\n//\nint64_t EMSwriteIndexMap(const int mmapID, EMSvalueType *key) {\n char *emsBuf = emsBufs[mmapID];\n volatile int64_t *bufInt64 = (int64_t *) emsBuf;\n volatile char *bufChar = emsBuf;\n volatile EMStag_t *bufTags = (EMStag_t *) emsBuf;\n volatile double *bufDouble = (double *) emsBuf;\n EMStag_t mapTags;\n\n // If the key already exists, use it\n int64_t idx = EMSkey2index(emsBuf, key, EMSisMapped);\n if(idx > 0) {\n // fprintf(stderr, \"write index map -- key already existed\\n\");\n return idx;\n }\n idx = EMSkey2index(emsBuf, key, false);\n int nTries = 0;\n if (EMSisMapped) {\n int matched = false;\n while (nTries < MAX_OPEN_HASH_STEPS && !matched) {\n idx = idx % bufInt64[EMScbData(EMS_ARR_NELEM)];\n // Wait until the map key is FULL, mark it busy while map lookup is performed\n mapTags.byte = EMStransitionFEtag(&bufTags[EMSmapTag(idx)], NULL, EMS_TAG_FULL, EMS_TAG_BUSY, EMS_TAG_ANY);\n mapTags.tags.fe = EMS_TAG_FULL; // When written back, mark FULL\n if (mapTags.tags.type == key->type || mapTags.tags.type == EMS_TYPE_UNDEFINED) {\n switch (mapTags.tags.type) {\n case EMS_TYPE_BOOLEAN:\n if ((int64_t) key->value == (bufInt64[EMSmapData(idx)] != 0)) {\n matched = true;\n bufTags[EMSmapTag(idx)].tags.fe = EMS_TAG_FULL;\n }\n break;\n case EMS_TYPE_INTEGER:\n if ((int64_t) key->value == bufInt64[EMSmapData(idx)]) {\n matched = true;\n bufTags[EMSmapTag(idx)].tags.fe = EMS_TAG_FULL;\n }\n break;\n case EMS_TYPE_FLOAT: {\n EMSulong_double alias;\n alias.u64 = (uint64_t) key->value;\n if (alias.d == bufDouble[EMSmapData(idx)]) {\n matched = true;\n bufTags[EMSmapTag(idx)].tags.fe = EMS_TAG_FULL;\n }\n }\n break;\n case EMS_TYPE_STRING: {\n int64_t keyStrOffset = bufInt64[EMSmapData(idx)];\n if (strcmp((const char *) key->value, (const char *) EMSheapPtr(keyStrOffset)) == 0) {\n matched = true;\n bufTags[EMSmapTag(idx)].tags.fe = EMS_TAG_FULL;\n }\n }\n break;\n case EMS_TYPE_UNDEFINED:\n // This map key index is still unused, so there was no match and a new\n // mapped element must be allocated to perform the tag bit transitions upon\n bufTags[EMSmapTag(idx)].tags.type = key->type;\n switch (key->type) {\n case EMS_TYPE_BOOLEAN:\n bufInt64[EMSmapData(idx)] = ((int64_t) key->value != 0);\n break;\n case EMS_TYPE_INTEGER:\n bufInt64[EMSmapData(idx)] = (int64_t) key->value;\n break;\n case EMS_TYPE_FLOAT: {\n EMSulong_double alias;\n alias.u64 = (uint64_t) key->value;\n bufDouble[EMSmapData(idx)] = alias.d;\n }\n break;\n case EMS_TYPE_STRING: {\n int64_t textOffset;\n EMS_ALLOC(textOffset, key->length + 1, bufChar,\n \"EMSwriteIndexMap(string): out of memory to store string\", -1);\n bufInt64[EMSmapData(idx)] = textOffset;\n strcpy((char *) EMSheapPtr(textOffset), (const char *) key->value);\n }\n break;\n case EMS_TYPE_UNDEFINED:\n bufInt64[EMSmapData(idx)] = 0xdeadbeef;\n break;\n default:\n fprintf(stderr, \"EMS ERROR: EMSwriteIndexMap: unknown arg type\\n\");\n }\n matched = true;\n break;\n default:\n fprintf(stderr, \"EMS ERROR: EMSwriteIndexMap: Unknown tag type (%d) on map key\\n\", mapTags.tags.type);\n matched = true;\n }\n }\n\n if (!matched) {\n // No match so set this key map back to full and try the next entry\n bufTags[EMSmapTag(idx)].tags.fe = EMS_TAG_FULL;\n nTries++;\n idx++;\n }\n }\n } else { // Wasn't mapped, do bounds check\n if (idx < 0 || idx >= bufInt64[EMScbData(EMS_ARR_NELEM)]) {\n fprintf(stderr, \"Wasn't mapped do bounds check\\n\");\n idx = -1;\n }\n }\n\n if (nTries >= MAX_OPEN_HASH_STEPS) {\n idx = -1;\n fprintf(stderr, \"EMSwriteIndexMap ran out of key mappings (ntries=%d)\\n\", nTries);\n }\n\n return idx;\n}\n\n\n\n//==================================================================\n// Read EMS memory, enforcing Full/Empty tag transitions\nbool EMSreadUsingTags(const int mmapID,\n EMSvalueType *key, // Index to read from\n EMSvalueType *returnValue,\n unsigned char initialFE, // Block until F/E tags are this value\n unsigned char finalFE) // Set the tag to this value when done\n{\n RESET_NAP_TIME;\n void *emsBuf = emsBufs[mmapID];\n volatile EMStag_t *bufTags = (EMStag_t *) emsBuf;\n volatile int64_t *bufInt64 = (int64_t *) emsBuf;\n volatile double *bufDouble = (double *) emsBuf;\n const char *bufChar = (const char *) emsBuf;\n\n returnValue->type = EMS_TYPE_UNDEFINED;\n returnValue->value = (void *) 0xdeafbeef; // TODO: Should return default value even when not doing write allocate\n\n EMStag_t newTag, oldTag, memTag;\n int64_t idx = EMSkey2index(emsBuf, key, EMSisMapped);\n\n // Allocate on Write, writes include modification of the tag:\n // If the EMS object being read is undefined and we're changing the f/e state\n // then allocate the undefined object and set the state. If the state is\n // not changing, do not allocate the undefined element.\n if(EMSisMapped && idx < 0) {\n if (finalFE != EMS_TAG_ANY) {\n idx = EMSwriteIndexMap(mmapID, key);\n if (idx < 0) {\n fprintf(stderr, \"EMSreadUsingTags: Unable to allocate on read for new map index\\n\");\n return false;\n }\n } else {\n return true;\n }\n }\n\n if (idx < 0 || idx >= bufInt64[EMScbData(EMS_ARR_NELEM)]) {\n fprintf(stderr, \"EMSreadUsingTags: index out of bounds\\n\");\n return false;\n }\n\n while (true) {\n memTag.byte = bufTags[EMSdataTag(idx)].byte;\n // Wait until FE tag is not FULL\n if (initialFE == EMS_TAG_ANY ||\n (initialFE != EMS_TAG_RW_LOCK && memTag.tags.fe == initialFE) ||\n (initialFE == EMS_TAG_RW_LOCK &&\n ((memTag.tags.fe == EMS_TAG_RW_LOCK && newTag.tags.rw < EMS_RW_NREADERS_MAX) ||\n memTag.tags.fe == EMS_TAG_FULL) &&\n (memTag.tags.rw < ((1 << EMS_TYPE_NBITS_RW) - 1)) // Counter is already saturated\n )\n ) {\n newTag.byte = memTag.byte;\n oldTag.byte = memTag.byte;\n newTag.tags.fe = EMS_TAG_BUSY;\n if (initialFE == EMS_TAG_RW_LOCK) {\n newTag.tags.rw++;\n } else {\n oldTag.tags.fe = initialFE;\n }\n // Transition FE from FULL to BUSY\n if (initialFE == EMS_TAG_ANY ||\n __sync_bool_compare_and_swap(&(bufTags[EMSdataTag(idx)].byte), oldTag.byte, newTag.byte)) {\n // Under BUSY lock:\n // Read the data, then reset the FE tag, then return the original value in memory\n newTag.tags.fe = finalFE;\n returnValue->type = newTag.tags.type;\n switch (newTag.tags.type) {\n case EMS_TYPE_BOOLEAN: {\n returnValue->value = (void *) (bufInt64[EMSdataData(idx)] != 0);\n if (finalFE != EMS_TAG_ANY) bufTags[EMSdataTag(idx)].byte = newTag.byte;\n if (EMSisMapped) bufTags[EMSmapTag(idx)].tags.fe = EMS_TAG_FULL;\n return true;\n }\n case EMS_TYPE_INTEGER: {\n returnValue->value = (void *) bufInt64[EMSdataData(idx)];\n if (finalFE != EMS_TAG_ANY) bufTags[EMSdataTag(idx)].byte = newTag.byte;\n if (EMSisMapped) bufTags[EMSmapTag(idx)].tags.fe = EMS_TAG_FULL;\n return true;\n }\n case EMS_TYPE_FLOAT: {\n EMSulong_double alias;\n alias.d = bufDouble[EMSdataData(idx)];\n returnValue->value = (void *) alias.u64;\n if (finalFE != EMS_TAG_ANY) bufTags[EMSdataTag(idx)].byte = newTag.byte;\n if (EMSisMapped) bufTags[EMSmapTag(idx)].tags.fe = EMS_TAG_FULL;\n return true;\n }\n case EMS_TYPE_JSON:\n case EMS_TYPE_STRING: {\n returnValue->value = (void *) EMSheapPtr(bufInt64[EMSdataData(idx)]);\n returnValue->length = strlen((const char *)returnValue->value);\n if (finalFE != EMS_TAG_ANY) bufTags[EMSdataTag(idx)].byte = newTag.byte;\n if (EMSisMapped) bufTags[EMSmapTag(idx)].tags.fe = EMS_TAG_FULL;\n return true;\n }\n case EMS_TYPE_UNDEFINED: {\n returnValue->value = (void *) 0xcafebeef;\n if (finalFE != EMS_TAG_ANY) bufTags[EMSdataTag(idx)].byte = newTag.byte;\n if (EMSisMapped) bufTags[EMSmapTag(idx)].tags.fe = EMS_TAG_FULL;\n return true;\n }\n default:\n fprintf(stderr, \"EMSreadUsingTags: unknown type (%d) read from memory\\n\", newTag.tags.type);\n return false;\n }\n } else {\n // Tag was marked BUSY between test read and CAS, must retry\n }\n } else {\n // Tag was already marked BUSY, must retry\n }\n // CAS failed or memory wasn't in initial state, wait and retry.\n // Permit preemptive map acquisition while waiting for data.\n if (EMSisMapped) { bufTags[EMSmapTag(idx)].tags.fe = EMS_TAG_FULL; }\n NANOSLEEP;\n if (EMSisMapped) {\n EMStransitionFEtag(&bufTags[EMSmapTag(idx)], NULL, EMS_TAG_FULL, EMS_TAG_BUSY, EMS_TAG_ANY);\n }\n }\n}\n\n\n//==================================================================\n// Read under multiple readers-single writer lock\nbool EMSreadRW(const int mmapID, EMSvalueType *key, EMSvalueType *returnValue) {\n return EMSreadUsingTags(mmapID, key, returnValue, EMS_TAG_RW_LOCK, EMS_TAG_RW_LOCK);\n}\n\n\n//==================================================================\n// Read when full and leave empty\nbool EMSreadFE(const int mmapID, EMSvalueType *key, EMSvalueType *returnValue) {\n return EMSreadUsingTags(mmapID, key, returnValue, EMS_TAG_FULL, EMS_TAG_EMPTY);\n}\n\n\n//==================================================================\n// Read when full and leave Full\nbool EMSreadFF(const int mmapID, EMSvalueType *key, EMSvalueType *returnValue) {\n return EMSreadUsingTags(mmapID, key, returnValue, EMS_TAG_FULL, EMS_TAG_FULL);\n}\n\n\n//==================================================================\n// Wrapper around read\nbool EMSread(const int mmapID, EMSvalueType *key, EMSvalueType *returnValue) {\n return EMSreadUsingTags(mmapID, key, returnValue, EMS_TAG_ANY, EMS_TAG_ANY);\n}\n\n\n//==================================================================\n// Decrement the reference count of the multiple readers-single writer lock\nint EMSreleaseRW(const int mmapID, EMSvalueType *key) {\n RESET_NAP_TIME;\n void *emsBuf = emsBufs[mmapID];\n volatile int64_t *bufInt64 = (int64_t *) emsBuf;\n volatile EMStag_t *bufTags = (EMStag_t *) emsBuf;\n EMStag_t newTag, oldTag;\n int64_t idx = EMSkey2index(emsBuf, key, EMSisMapped);\n if (idx < 0 || idx >= bufInt64[EMScbData(EMS_ARR_NELEM)]) {\n fprintf(stderr, \"EMSreleaseRW: invalid index (%\" PRIi64 \")\\n\", idx);\n return -1;\n }\n\n while (true) {\n oldTag.byte = bufTags[EMSdataTag(idx)].byte;\n newTag.byte = oldTag.byte;\n if (oldTag.tags.fe == EMS_TAG_RW_LOCK) {\n // Already under a RW lock\n if (oldTag.tags.rw == 0) {\n // Assert the RW count is consistent with the lock state\n fprintf(stderr, \"EMSreleaseRW: locked but Count already 0\\n\");\n return -1;\n } else {\n // Decrement the RW reference count\n newTag.tags.rw--;\n // If this is the last reader, set the FE tag back to full\n if (newTag.tags.rw == 0) { newTag.tags.fe = EMS_TAG_FULL; }\n // Attempt to commit the RW reference count & FE tag\n if (__sync_bool_compare_and_swap(&(bufTags[EMSdataTag(idx)].byte), oldTag.byte, newTag.byte)) {\n return (int) newTag.tags.rw;\n } else {\n // Another thread decremented the RW count while we also tried\n }\n }\n } else {\n if (oldTag.tags.fe != EMS_TAG_BUSY) {\n // Assert the RW lock being release is not in some other state then RW_LOCK or BUSY\n fprintf(stderr, \"EMSreleaseRW: The RW lock being released is in some other state then RW_LOCK or BUSY\\n\");\n return -1;\n }\n }\n // Failed to update the RW count, sleep and retry\n NANOSLEEP;\n }\n}\n\n\n//==================================================================\n// Write EMS honoring the F/E tags\nbool EMSwriteUsingTags(int mmapID,\n EMSvalueType *key,\n EMSvalueType *value,\n unsigned char initialFE, // Block until F/E tags are this value\n unsigned char finalFE) // Set the tag to this value when done\n{\n RESET_NAP_TIME;\n char *emsBuf = emsBufs[mmapID];\n volatile EMStag_t *bufTags = (EMStag_t *) emsBuf;\n volatile int64_t *bufInt64 = (int64_t *) emsBuf;\n volatile double *bufDouble = (double *) emsBuf;\n char *bufChar = emsBuf;\n EMStag_t newTag, oldTag, memTag;\n int64_t idx = EMSwriteIndexMap(mmapID, key);\n if (idx < 0) {\n fprintf(stderr, \"EMSwriteUsingTags: index out of bounds\\n\");\n return false;\n }\n\n // Wait for the memory to be in the initial F/E state and transition to Busy\n if (initialFE != EMS_TAG_ANY) {\n volatile EMStag_t *maptag;\n if (EMSisMapped) { maptag = &bufTags[EMSmapTag(idx)]; }\n else { maptag = NULL; }\n EMStransitionFEtag(&bufTags[EMSdataTag(idx)], maptag,\n initialFE, EMS_TAG_BUSY, EMS_TAG_ANY);\n }\n\n while (true) {\n idx = idx % bufInt64[EMScbData(EMS_ARR_NELEM)];\n memTag.byte = bufTags[EMSdataTag(idx)].byte;\n // Wait until FE tag is not BUSY\n if (initialFE != EMS_TAG_ANY || finalFE == EMS_TAG_ANY || memTag.tags.fe != EMS_TAG_BUSY) {\n oldTag.byte = memTag.byte;\n newTag.byte = memTag.byte;\n if (finalFE != EMS_TAG_ANY) newTag.tags.fe = EMS_TAG_BUSY;\n // Transition FE from !BUSY to BUSY\n if (initialFE != EMS_TAG_ANY || finalFE == EMS_TAG_ANY ||\n __sync_bool_compare_and_swap(&(bufTags[EMSdataTag(idx)].byte), oldTag.byte, newTag.byte)) {\n // If the old data was a string, free it because it will be overwritten\n if (oldTag.tags.type == EMS_TYPE_STRING || oldTag.tags.type == EMS_TYPE_JSON) {\n EMS_FREE(bufInt64[EMSdataData(idx)]);\n }\n\n // Store argument value into EMS memory\n switch (value->type) {\n case EMS_TYPE_BOOLEAN:\n bufInt64[EMSdataData(idx)] = (int64_t) value->value;\n break;\n case EMS_TYPE_INTEGER:\n bufInt64[EMSdataData(idx)] = (int64_t) value->value;\n break;\n case EMS_TYPE_FLOAT: {\n EMSulong_double alias;\n alias.u64 = (uint64_t) value->value;\n bufDouble[EMSdataData(idx)] = alias.d;\n }\n break;\n case EMS_TYPE_JSON:\n case EMS_TYPE_STRING: {\n int64_t textOffset;\n EMS_ALLOC(textOffset, value->length + 1, bufChar, // NULL padding at end\n \"EMSwriteUsingTags: out of memory to store string\", false);\n bufInt64[EMSdataData(idx)] = textOffset;\n strcpy(EMSheapPtr(textOffset), (const char *) value->value);\n }\n break;\n case EMS_TYPE_UNDEFINED:\n bufInt64[EMSdataData(idx)] = 0xdeadbeef;\n break;\n default:\n fprintf(stderr, \"EMSwriteUsingTags: Unknown arg type\\n\");\n return false;\n }\n\n oldTag.byte = newTag.byte;\n if (finalFE != EMS_TAG_ANY) {\n newTag.tags.fe = finalFE;\n newTag.tags.rw = 0;\n }\n newTag.tags.type = value->type;\n if (finalFE != EMS_TAG_ANY && bufTags[EMSdataTag(idx)].byte != oldTag.byte) {\n fprintf(stderr, \"EMSwriteUsingTags: Lost tag lock while BUSY\\n\");\n return false;\n }\n\n // Set the tags for the data (and map, if used) back to full to finish the operation\n bufTags[EMSdataTag(idx)].byte = newTag.byte;\n if (EMSisMapped) bufTags[EMSmapTag(idx)].tags.fe = EMS_TAG_FULL;\n return true;\n } else {\n // Tag was marked BUSY between test read and CAS, must retry\n }\n } else {\n // Tag was already marked BUSY, must retry\n }\n // Failed to set the tags, sleep and retry\n NANOSLEEP;\n }\n}\n\n\n//==================================================================\n// WriteXF\nbool EMSwriteXF(int mmapID, EMSvalueType *key, EMSvalueType *value) {\n return EMSwriteUsingTags(mmapID, key, value, EMS_TAG_ANY, EMS_TAG_FULL);\n}\n\n//==================================================================\n// WriteXE\nbool EMSwriteXE(int mmapID, EMSvalueType *key, EMSvalueType *value) {\n return EMSwriteUsingTags(mmapID, key, value, EMS_TAG_ANY, EMS_TAG_EMPTY);\n}\n\n//==================================================================\n// WriteEF\nbool EMSwriteEF(int mmapID, EMSvalueType *key, EMSvalueType *value) {\n return EMSwriteUsingTags(mmapID, key, value, EMS_TAG_EMPTY, EMS_TAG_FULL);\n}\n\n//==================================================================\n// Write\nbool EMSwrite(int mmapID, EMSvalueType *key, EMSvalueType *value) {\n return EMSwriteUsingTags(mmapID, key, value, EMS_TAG_ANY, EMS_TAG_ANY);\n}\n\n\n//==================================================================\n// Set only the Full/Empty tag from JavaScript \n// without inspecting or modifying the data.\nbool EMSsetTag(int mmapID, EMSvalueType *key, bool is_full) {\n void *emsBuf = emsBufs[mmapID];\n volatile EMStag_t *bufTags = (EMStag_t *) emsBuf;\n volatile int64_t *bufInt64 = (int64_t *) emsBuf;\n EMStag_t tag;\n\n int64_t idx = EMSkey2index(emsBuf, key, EMSisMapped);\n if (idx < 0 || idx >= bufInt64[EMScbData(EMS_ARR_NELEM)]) {\n return false;\n }\n\n tag.byte = bufTags[EMSdataTag(idx)].byte;\n if (is_full) {\n tag.tags.fe = EMS_TAG_FULL;\n } else {\n tag.tags.fe = EMS_TAG_EMPTY;\n }\n bufTags[EMSdataTag(idx)].byte = tag.byte;\n return true;\n}\n\n\n//==================================================================\n// Release all the resources associated with an EMS array\nbool EMSdestroy(int mmapID, bool do_unlink) {\n void *emsBuf = emsBufs[mmapID];\n if(munmap(emsBuf, emsBufLengths[mmapID]) != 0) {\n fprintf(stderr, \"EMSdestroy: Unable to unmap memory\\n\");\n return false;\n }\n\n if (do_unlink) {\n if (unlink(emsBufFilenames[mmapID]) != 0) {\n fprintf(stderr, \"EMSdestroy: Unable to unlink file\\n\");\n return false;\n }\n }\n\n emsBufFilenames[mmapID][0] = 0;\n emsBufLengths[mmapID] = 0;\n emsBufs[mmapID] = NULL;\n return true;\n}\n\n\n\n//==================================================================\n// Return the key of a mapped object given the EMS index\nbool EMSindex2key(int mmapID, int64_t idx, EMSvalueType *key) {\n void *emsBuf = emsBufs[mmapID];\n volatile int64_t *bufInt64 = (int64_t *) emsBuf;\n char *bufChar = (char *) emsBuf;\n volatile EMStag_t *bufTags = (EMStag_t *) emsBuf;\n volatile double *bufDouble = (double *) emsBuf;\n\n if(!EMSisMapped) {\n fprintf(stderr, \"EMSindex2key: Unmapping an index but Array is not mapped\\n\");\n return false;\n }\n\n if (idx < 0 || idx >= bufInt64[EMScbData(EMS_ARR_NELEM)]) {\n fprintf(stderr, \"EMSindex2key: index out of bounds\\n\");\n return false;\n }\n\n key->type = bufTags[EMSmapTag(idx)].tags.type;\n switch (key->type) {\n case EMS_TYPE_BOOLEAN:\n case EMS_TYPE_INTEGER: {\n key->value = (void *) bufInt64[EMSmapData(idx)];\n return true;\n }\n case EMS_TYPE_FLOAT: {\n EMSulong_double alias;\n alias.d = bufDouble[EMSmapData(idx)];\n key->value = (void *) alias.u64;\n return true;\n }\n case EMS_TYPE_JSON:\n case EMS_TYPE_STRING: {\n key->value = (void *)(EMSheapPtr(bufInt64[EMSmapData(idx)]));\n return true;\n }\n case EMS_TYPE_UNDEFINED: {\n key->value = NULL;\n return true;\n }\n default:\n fprintf(stderr, \"EMSindex2key unknown type\\n\");\n return false;\n }\n}\n\n\n//==================================================================\n// Synchronize the EMS memory to persistent storage\n//\nbool EMSsync(int mmapID) {\n // resultIdx = msync((void*) emsBuf, pgsize, flags);\n printf(\"EMSsync() was called but stubbed out\\n\");\n return false;\n}\n\n\n//==================================================================\n// EMS Entry Point: Allocate and initialize the EMS domain memory\n//\nint EMSinitialize(int64_t nElements, // 0\n size_t heapSize, // 1\n bool useMap, // 2\n const char *filename, // 3\n bool persist, // 4\n bool useExisting, // 5\n bool doDataFill, // 6\n bool fillIsJSON, // 7\n EMSvalueType *fillValue,// 8\n bool doSetFEtags, // 9\n bool setFEtagsFull, // 10\n int EMSmyIDarg, // 11\n bool pinThreads, // 12\n int32_t nThreads, // 13\n int32_t pctMLock ) { // 14\n int fd;\n EMSmyID = EMSmyIDarg;\n\n // Node 0 is first and always has mutual exclusion during initialization\n // perform once-only initialization here\n if (EMSmyID == 0) {\n if (!useExisting) {\n unlink(filename);\n shm_unlink(filename);\n }\n }\n\n if (useExisting) {\n struct timespec sleep_time;\n sleep_time.tv_sec = 0;\n sleep_time.tv_nsec = 200000000;\n struct stat statbuf;\n while (stat(filename, &statbuf) != 0) nanosleep(&sleep_time, NULL); // TODO: timeout?\n }\n\n if (persist)\n fd = open(filename, O_APPEND | O_CREAT | O_RDWR, S_IRUSR | S_IWUSR);\n else\n fd = shm_open(filename, O_CREAT | O_RDWR, S_IRUSR | S_IWUSR);\n\n if (fd < 0) {\n perror(\"Error opening shared memory file -- Possibly need to be root?\");\n return -1;\n }\n\n size_t nMemBlocks = (heapSize / EMS_MEM_BLOCKSZ) + 1;\n size_t nMemBlocksPow2 = emsNextPow2((int64_t) nMemBlocks);\n int32_t nMemLevels = __builtin_ctzl(nMemBlocksPow2);\n size_t bottomOfMalloc;\n size_t filesize;\n\n size_t bottomOfMap = (size_t)EMSdataTagWord(nElements) + (size_t)EMSwordSize; // Map begins 1 word AFTER the last tag word of data\n if (useMap) {\n bottomOfMalloc = bottomOfMap + bottomOfMap;\n } else {\n bottomOfMalloc = bottomOfMap;\n }\n size_t bottomOfHeap = bottomOfMalloc + sizeof(struct emsMem) + (nMemBlocksPow2 * 2 - 2);\n\n if (nElements <= 0) {\n filesize = EMS_CB_LOCKS + nThreads; // EMS Control Block\n filesize *= sizeof(int);\n } else {\n filesize = bottomOfHeap + (nMemBlocksPow2 * EMS_MEM_BLOCKSZ);\n }\n if (ftruncate(fd, (off_t) filesize) != 0) {\n if (errno != EINVAL) {\n fprintf(stderr, \"EMSinitialize: Error during initialization, unable to set memory size to %\" PRIu64 \" bytes\\n\",\n (uint64_t) filesize);\n return -1;\n }\n }\n\n char *emsBuf = (char *) mmap(0, filesize, PROT_READ | PROT_WRITE, MAP_SHARED, fd, (off_t) 0);\n if (emsBuf == MAP_FAILED) {\n fprintf(stderr, \"EMSinitialize: Unable to map domain memory\\n\");\n return -1;\n }\n close(fd);\n\n if (nElements <= 0) pctMLock = 100; // lock RAM if master control block\n if (mlock((void *) emsBuf, (size_t) (filesize * (pctMLock / 100))) != 0) {\n fprintf(stderr, \"EMSinitialize NOTICE: EMS thread %d was not able to lock EMS memory to RAM for %s\\n\", EMSmyID, filename);\n } else {\n // success\n }\n\n volatile int64_t *bufInt64 = (int64_t *) emsBuf;\n volatile double *bufDouble = (double *) emsBuf;\n char *bufChar = emsBuf;\n volatile int *bufInt32 = (int32_t *) emsBuf;\n volatile EMStag_t *bufTags = (EMStag_t *) emsBuf;\n\n if (EMSmyID == 0) {\n if (nElements <= 0) { // This is the EMS CB\n bufInt32[EMS_CB_NTHREADS] = nThreads;\n bufInt32[EMS_CB_NBAR0] = nThreads;\n bufInt32[EMS_CB_NBAR1] = nThreads;\n bufInt32[EMS_CB_BARPHASE] = 0;\n bufInt32[EMS_CB_CRITICAL] = 0;\n bufInt32[EMS_CB_SINGLE] = 0;\n for (int i = EMS_CB_LOCKS; i < EMS_CB_LOCKS + nThreads; i++) {\n bufInt32[i] = EMS_TAG_BUSY; // Reset all locks\n }\n } else { // This is a user data domain\n if (!useExisting) {\n EMStag_t tag;\n tag.tags.rw = 0;\n tag.tags.type = EMS_TYPE_INTEGER;\n tag.tags.fe = EMS_TAG_FULL;\n bufInt64[EMScbData(EMS_ARR_NELEM)] = nElements;\n bufInt64[EMScbData(EMS_ARR_HEAPSZ)] = heapSize; // Unused?\n bufInt64[EMScbData(EMS_ARR_MAPBOT)] = bottomOfMap / EMSwordSize;\n bufInt64[EMScbData(EMS_ARR_MALLOCBOT)] = bottomOfMalloc;\n bufInt64[EMScbData(EMS_ARR_HEAPBOT)] = bottomOfHeap;\n bufInt64[EMScbData(EMS_ARR_Q_BOTTOM)] = 0;\n bufTags[EMScbTag(EMS_ARR_Q_BOTTOM)].byte = tag.byte;\n bufInt64[EMScbData(EMS_ARR_STACKTOP)] = 0;\n bufTags[EMScbTag(EMS_ARR_STACKTOP)].byte = tag.byte;\n bufInt64[EMScbData(EMS_ARR_MEM_MUTEX)] = EMS_TAG_EMPTY;\n bufInt64[EMScbData(EMS_ARR_FILESZ)] = filesize;\n struct emsMem *emsMemBuffer = (struct emsMem *) &bufChar[bufInt64[EMScbData(EMS_ARR_MALLOCBOT)]];\n emsMemBuffer->level = nMemLevels;\n }\n }\n }\n\n if (pinThreads) {\n#if defined(__linux)\n cpu_set_t cpuset;\n CPU_ZERO(&cpuset);\n CPU_SET((EMSmyID % nThreads), &cpuset); // Round-robin over-subscribed systems\n sched_setaffinity(0, sizeof(cpu_set_t), &cpuset);\n#endif\n }\n EMStag_t tag;\n tag.tags.rw = 0;\n int64_t iterPerThread = (nElements / nThreads) + 1;\n int64_t startIter = iterPerThread * EMSmyID;\n int64_t endIter = iterPerThread * (EMSmyID + 1);\n size_t fillStrLen = 0;\n if (doDataFill && (fillValue->type == EMS_TYPE_JSON || fillValue->type == EMS_TYPE_STRING)) {\n fillStrLen = fillValue->length;\n }\n if (endIter > nElements) endIter = nElements;\n for (int64_t idx = startIter; idx < endIter; idx++) {\n tag.tags.rw = 0;\n if (doDataFill) {\n tag.tags.type = fillValue->type;\n switch (tag.tags.type) {\n case EMS_TYPE_BOOLEAN:\n case EMS_TYPE_INTEGER:\n bufInt64[EMSdataData(idx)] = (int64_t) fillValue->value;\n break;\n case EMS_TYPE_FLOAT: {\n EMSulong_double alias;\n alias.u64 = (uint64_t) fillValue->value;\n bufDouble[EMSdataData(idx)] = alias.d;\n }\n break;\n case EMS_TYPE_UNDEFINED:\n bufInt64[EMSdataData(idx)] = 0xdeadbeef;\n break;\n case EMS_TYPE_JSON:\n case EMS_TYPE_STRING: {\n int64_t textOffset;\n EMS_ALLOC(textOffset, fillStrLen + 1, bufChar,\n \"EMSinitialize: out of memory to store string\", false);\n bufInt64[EMSdataData(idx)] = textOffset;\n strcpy(EMSheapPtr(textOffset), (const char *) fillValue->value);\n }\n break;\n default:\n fprintf(stderr, \"EMSinitialize: fill type is unknown\\n\");\n return -1;\n }\n } else {\n tag.tags.type = EMS_TYPE_UNDEFINED;\n }\n\n if (doSetFEtags) {\n if (setFEtagsFull) tag.tags.fe = EMS_TAG_FULL;\n else tag.tags.fe = EMS_TAG_EMPTY;\n }\n\n if (doSetFEtags || doDataFill) {\n bufTags[EMSdataTag(idx)].byte = tag.byte;\n }\n\n if (useMap) {\n bufTags[EMSmapTag(idx)].tags.fe = EMS_TAG_FULL;\n if (!useExisting) { bufTags[EMSmapTag(idx)].tags.type = EMS_TYPE_UNDEFINED; }\n }\n }\n\n int emsBufN = 0;\n while(emsBufN < EMS_MAX_N_BUFS && emsBufs[emsBufN] != NULL) emsBufN++;\n if(emsBufN < EMS_MAX_N_BUFS) {\n emsBufs[emsBufN] = emsBuf;\n emsBufLengths[emsBufN] = filesize;\n strncpy(emsBufFilenames[emsBufN], filename, MAX_FNAME_LEN);\n } else {\n fprintf(stderr, \"EMSinitialize: ERROR - Unable to allocate a buffer ID/index\\n\");\n emsBufN = -1;\n }\n\n return emsBufN;\n}\n\n"
},
{
"path": "src/ems.h",
"content": "/*-----------------------------------------------------------------------------+\n | Extended Memory Semantics (EMS) Version 1.4.0 |\n | Synthetic Semantics http://www.synsem.com/ mogill@synsem.com |\n +-----------------------------------------------------------------------------+\n | Copyright (c) 2011-2014, Synthetic Semantics LLC. All rights reserved. |\n | Copyright (c) 2015-2016, Jace A Mogill. All rights reserved. |\n | |\n | Redistribution and use in source and binary forms, with or without |\n | modification, are permitted provided that the following conditions are met: |\n | * Redistributions of source code must retain the above copyright |\n | notice, this list of conditions and the following disclaimer. |\n | * Redistributions in binary form must reproduce the above copyright |\n | notice, this list of conditions and the following disclaimer in the |\n | documentation and/or other materials provided with the distribution. |\n | * Neither the name of the Synthetic Semantics nor the names of its |\n | contributors may be used to endorse or promote products derived |\n | from this software without specific prior written permission. |\n | |\n | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS |\n | \"AS IS\" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT |\n | LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR |\n | A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL SYNTHETIC |\n | SEMANTICS LLC BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, |\n | EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, |\n | PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR |\n | PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF |\n | LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING |\n | NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS |\n | SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. |\n | |\n +-----------------------------------------------------------------------------*/\n#ifndef EMSPROJ_EMS_H\n#define EMSPROJ_EMS_H\n#define __STDC_FORMAT_MACROS 1\n#include \n#include \n#include \n#include \n#include \n#include \n#include \n#include \n#include \n#include \n#include \n#include \n\n#if !defined _GNU_SOURCE\n# define _GNU_SOURCE\n#endif\n#include \n\n#include \"ems_alloc.h\"\n\n//==================================================================\n// EMS Full/Empty Tag States\n//\n#define EMS_TAG_ANY ((unsigned char)4) // Never stored, used for matching\n#define EMS_TAG_RW_LOCK ((unsigned char)3)\n#define EMS_TAG_BUSY ((unsigned char)2)\n#define EMS_TAG_EMPTY ((unsigned char)1)\n#define EMS_TAG_FULL ((unsigned char)0)\n\n\n//==================================================================\n// EMS Data types\n//\n#define EMS_TYPE_INVALID ((unsigned char)0)\n#define EMS_TYPE_BOOLEAN ((unsigned char)1)\n#define EMS_TYPE_STRING ((unsigned char)2)\n#define EMS_TYPE_FLOAT ((unsigned char)3)\n#define EMS_TYPE_INTEGER ((unsigned char)4)\n#define EMS_TYPE_UNDEFINED ((unsigned char)5)\n#define EMS_TYPE_JSON ((unsigned char)6) // Catch-all for JSON arrays and Objects\n\n\n\n//==================================================================\n// Control Block layout stored at the head of each EMS array\n//\n// Name Offset Description of contents\n//---------------------------------------------------------------------------------------\n#define NWORDS_PER_CACHELINE 16\n#define EMS_ARR_NELEM (0 * NWORDS_PER_CACHELINE) // Maximum number of elements in the EMS array\n#define EMS_ARR_HEAPSZ (1 * NWORDS_PER_CACHELINE) // # bytes of storage for array data: strings, JSON, maps, etc.\n#define EMS_ARR_Q_BOTTOM (2 * NWORDS_PER_CACHELINE) // Current index of the queue bottom\n#define EMS_ARR_STACKTOP (3 * NWORDS_PER_CACHELINE) // Current index of the top of the stack/queue\n#define EMS_ARR_MAPBOT (4 * NWORDS_PER_CACHELINE) // Index of the base of the index map\n#define EMS_ARR_MALLOCBOT (5 * NWORDS_PER_CACHELINE) // Index of the base of the heap -- malloc structs start here\n#define EMS_ARR_HEAPBOT (6 * NWORDS_PER_CACHELINE) // Index of the base of data on the heap -- strings start here\n#define EMS_ARR_MEM_MUTEX (7 * NWORDS_PER_CACHELINE) // Mutex lock for thememory allocator of this EMS region's\n#define EMS_ARR_FILESZ (8 * NWORDS_PER_CACHELINE) // Total size in bytes of the EMS region\n// Tag data may follow data by as much as 8 words, so\n// A gap of at least 8 words is required to leave space for\n// the tags associated with header data\n#define EMS_ARR_CB_SIZE (16 * NWORDS_PER_CACHELINE) // Index of the first EMS array element\n\n\n\n//==================================================================\n// EMS Control Block -- Global State for EMS\n#define EMS_CB_NTHREADS 0 // Number of threads\n#define EMS_CB_NBAR0 1 // Number of threads at Barrier 0\n#define EMS_CB_NBAR1 2 // Number of threads at Barrier 1\n#define EMS_CB_BARPHASE 3 // Current Barrier Phase (0 or 1)\n#define EMS_CB_CRITICAL 4 // Mutex for critical regions\n#define EMS_CB_SINGLE 5 // Number of threads passed through an execute-once region\n#define EMS_LOOP_IDX 6 // Index of next iteration in a Parallel loop to schedule\n#define EMS_LOOP_START 7 // Index of first iteration in a parallel loop\n#define EMS_LOOP_END 8 // Index of last iteration in a parallel loop\n#define EMS_LOOP_CHUNKSZ 9 // Current Number of iterations per thread\n#define EMS_LOOP_MINCHUNK 10 // Smallest number of iterations per thread\n#define EMS_LOOP_SCHED 11 // Parallel loop scheduling method:\n#define EMS_SCHED_GUIDED 1200\n#define EMS_SCHED_DYNAMIC 1201\n#define EMS_SCHED_STATIC 1202\n#define EMS_CB_LOCKS 12 // First index of an array of locks, one lock per thread\n\n\n\n//==================================================================\n// Pointers to mmapped EMS buffers\n#define EMS_MAX_N_BUFS 4096\n#define MAX_NUMBER2STR_LEN 40 // Maximum number of characters in a %d or %f format\n#define MAX_FNAME_LEN 256\nextern char *emsBufs[EMS_MAX_N_BUFS];\nextern size_t emsBufLengths[EMS_MAX_N_BUFS];\nextern char emsBufFilenames[EMS_MAX_N_BUFS][MAX_FNAME_LEN];\n\n// Maximum number of slots to check due to conflicts\n#define MAX_OPEN_HASH_STEPS 200\n\n\n//==================================================================\n// Macros to translate from EMS Data Indexes and EMS Control Block\n// indexes to offsets in the EMS shared memory\n#define EMSwordSize (sizeof(size_t))\n#define EMSnWordsPerTagWord (EMSwordSize-1)\n#define EMSnWordsPerLine EMSwordSize\n\n\n//==================================================================\n// Layout of EMS memory\n// Tagged Memory\n// CB: Control Block of array state\n// Data: Scalar user data\n// Map: Scalar index map data\n// Untagged Memory\n// Malloc: Storage for the free/used structures\n// Heap: Open Heap storage\n#define EMSappIdx2emsIdx(idx) ((((idx) / EMSnWordsPerTagWord) * EMSnWordsPerLine) + ((idx) % EMSnWordsPerTagWord) )\n#define EMSappIdx2LineIdx(idx) ( ((idx) / EMSnWordsPerTagWord) * EMSnWordsPerLine)\n#define EMSappIdx2TagWordIdx(idx) ( EMSappIdx2LineIdx(idx) + EMSnWordsPerTagWord )\n#define EMSappIdx2TagWordOffset(idx) ( EMSappIdx2TagWordIdx(idx) * EMSwordSize )\n#define EMSappTag2emsTag(idx) ( EMSappIdx2TagWordOffset(idx) + ((idx) % EMSnWordsPerTagWord) )\n#define EMScbData(idx) EMSappIdx2emsIdx(idx)\n#define EMScbTag(idx) EMSappTag2emsTag(idx)\n#define EMSdataData(idx) ( EMSappIdx2emsIdx((idx) + EMS_ARR_CB_SIZE) )\n#define EMSdataTag(idx) ( EMSappTag2emsTag((idx) + EMS_ARR_CB_SIZE) )\n#define EMSdataTagWord(idx) ( EMSappIdx2TagWordOffset((idx) + EMS_ARR_CB_SIZE) )\n#define EMSmapData(idx) ( EMSappIdx2emsIdx((idx) + EMS_ARR_CB_SIZE + bufInt64[EMScbData(EMS_ARR_NELEM)]) )\n#define EMSmapTag(idx) ( EMSappTag2emsTag((idx) + EMS_ARR_CB_SIZE + bufInt64[EMScbData(EMS_ARR_NELEM)]) )\n#define EMSheapPtr(idx) ( &bufChar[ bufInt64[EMScbData(EMS_ARR_HEAPBOT)] + (idx) ] )\n\n#define EMS_MEM_MALLOCBOT(bufChar) ((struct emsMem *) &bufChar[ bufInt64[EMScbData(EMS_ARR_MALLOCBOT)] ])\n\n\n//==================================================================\n// Yield the processor and sleep (using exponential decay) without\n// using resources/\n// Used within spin-loops to reduce hot-spotting\n#define RESET_NAP_TIME int EMScurrentNapTime = 1\n#define MAX_NAP_TIME 1000000\n#define NANOSLEEP { \\\n struct timespec sleep_time; \\\n sleep_time.tv_sec = 0; \\\n sleep_time.tv_nsec = EMScurrentNapTime; \\\n nanosleep(&sleep_time, NULL); \\\n EMScurrentNapTime *= 2; \\\n if(EMScurrentNapTime > MAX_NAP_TIME) { \\\n EMScurrentNapTime = MAX_NAP_TIME; \\\n } \\\n }\n\n\n#define EMS_ALLOC(addr, len, bufChar, errmsg, retval) \\\n addr = emsMutexMem_alloc( EMS_MEM_MALLOCBOT(bufChar), \\\n (size_t) len, (char*) &bufInt64[EMScbData(EMS_ARR_MEM_MUTEX)] ); \\\n if(addr < 0) { \\\n fprintf(stderr, \"%s:%d (%s) ERROR: EMS memory allocation of len(%zx) failed: %s\\n\", \\\n __FILE__, __LINE__, __FUNCTION__, len, errmsg); \\\n return retval; \\\n }\n\n#define EMS_FREE(addr) \\\n emsMutexMem_free( EMS_MEM_MALLOCBOT(bufChar), \\\n (size_t) addr, (char*) &bufInt64[EMScbData(EMS_ARR_MEM_MUTEX)] )\n\nsize_t emsMutexMem_alloc(struct emsMem *heap, // Base of EMS malloc structs\n size_t len, // Number of bytes to allocate\n volatile char *mutex); // Pointer to the mem allocator's mutex\nvoid emsMutexMem_free(struct emsMem *heap, // Base of EMS malloc structs\n size_t addr, // Offset of alloc'd block in EMS memory\n volatile char *mutex); // Pointer to the mem allocator's mutex\n\nextern int EMSmyID; // EMS Thread ID\n\n#define EMSisMapped (bufInt64[EMScbData(EMS_ARR_MAPBOT)]*(int64_t)EMSwordSize != bufInt64[EMScbData(EMS_ARR_MALLOCBOT)])\n\n#include \"ems_proto.h\"\n\n#endif //EMSPROJ_EMS_H\n"
},
{
"path": "src/ems_alloc.cc",
"content": "/*-----------------------------------------------------------------------------+\n | Extended Memory Semantics (EMS) Version 1.3.0 |\n | Synthetic Semantics http://www.synsem.com/ mogill@synsem.com |\n +-----------------------------------------------------------------------------+\n | Copyright (c) 2011-2014, Synthetic Semantics LLC. All rights reserved. |\n | Copyright (c) 2015-2016, Jace A Mogill. All rights reserved. |\n | |\n | Redistribution and use in source and binary forms, with or without |\n | modification, are permitted provided that the following conditions are met: |\n | * Redistributions of source code must retain the above copyright |\n | notice, this list of conditions and the following disclaimer. |\n | * Redistributions in binary form must reproduce the above copyright |\n | notice, this list of conditions and the following disclaimer in the |\n | documentation and/or other materials provided with the distribution. |\n | * Neither the name of the Synthetic Semantics nor the names of its |\n | contributors may be used to endorse or promote products derived |\n | from this software without specific prior written permission. |\n | |\n | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS |\n | \"AS IS\" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT |\n | LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR |\n | A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL SYNTHETIC |\n | SEMANTICS LLC BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, |\n | EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, |\n | PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR |\n | PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF |\n | LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING |\n | NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS |\n | SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. |\n | |\n +-----------------------------------------------------------------------------*/\n#include \"ems_alloc.h\"\n#include \n#include \n#include \n\n#define BUDDY_UNUSED 0\n#define BUDDY_USED 1\n#define BUDDY_SPLIT 2\n#define BUDDY_FULL 3\n\n\n//-----------------------------------------------------------------------------+\n// Allocate memory for testing -- \n// Performed as part of new EMS object initialization\nstruct emsMem *emsMem_new(int level) {\n size_t size = 1UL << level;\n printf(\"emsMem_new: malloc sz=%ld\\n\", size);\n struct emsMem *self = (struct emsMem *) malloc(sizeof(struct emsMem) + sizeof(uint8_t) * (size * 2 - 2));\n self->level = level;\n memset(self->tree, BUDDY_UNUSED, size * 2 - 1);\n return self;\n}\n\n//-----------------------------------------------------------------------------+\n// De-allocate memory from testing, not part of EMS object\nvoid emsMem_delete(struct emsMem *self) {\n free(self);\n}\n\n\n//-----------------------------------------------------------------------------+\n// Pow-2 utility functions for 64 bit\nsize_t emsNextPow2(int64_t x) {\n if (__builtin_popcountll(x) == 1) return x;\n // Yes, that will overflow on 63 bit numbers. Hopefully someone will rewrite this by then.\n // fprintf(stderr, \">>>lz=%d shift=%d\\n\", __builtin_clzl(x), (64 - __builtin_clzl(x)));\n return (1UL << (64 - __builtin_clzll(x)));\n}\n\n\nstatic inline int64_t EMS_index_offset(int64_t index, int32_t level, int64_t max_level) {\n return ((index + 1) - (1UL << level)) << (max_level - level);\n}\n\n\n//-----------------------------------------------------------------------------+\n// Mark the parent buddy of this buddy\nstatic void EMS_mark_parent(struct emsMem *self, int64_t index) {\n for (; ;) {\n int64_t buddy = index - 1 + (index & 1) * 2;\n if (buddy > 0 && (self->tree[buddy] == BUDDY_USED || self->tree[buddy] == BUDDY_FULL)) {\n index = (index + 1) / 2 - 1;\n self->tree[index] = BUDDY_FULL;\n } else {\n return;\n }\n }\n}\n\n\n//-----------------------------------------------------------------------------+\n// Allocate new memory from the EMS heap\nsize_t emsMem_alloc(struct emsMem *self, size_t bytesRequested) {\n size_t size;\n size = emsNextPow2((bytesRequested + (EMS_MEM_BLOCKSZ - 1)) / EMS_MEM_BLOCKSZ);\n if (size == 0) size++;\n size_t length = 1UL << self->level;\n\n // fprintf(stderr, \"emsMem_alloc: self=%x size=%ld s=%ld len=%ld\\n\", self, size, s, length);\n if (size > length) return -1;\n\n int64_t index = 0;\n int32_t level = 0;\n\n while (index >= 0) {\n if (size == length) {\n if (self->tree[index] == BUDDY_UNUSED) {\n self->tree[index] = BUDDY_USED;\n EMS_mark_parent(self, index);\n return ((size_t)EMS_index_offset(index, level, self->level) * EMS_MEM_BLOCKSZ);\n }\n } else {\n // size < length\n switch (self->tree[index]) {\n case BUDDY_USED:\n case BUDDY_FULL:\n break;\n case BUDDY_UNUSED:\n // split first\n self->tree[index] = BUDDY_SPLIT;\n self->tree[index * 2 + 1] = BUDDY_UNUSED;\n self->tree[index * 2 + 2] = BUDDY_UNUSED;\n default:\n index = index * 2 + 1;\n length /= 2;\n level++;\n continue;\n }\n }\n if (index & 1) {\n ++index;\n continue;\n }\n for (; ;) {\n level--;\n length *= 2;\n index = (index + 1) / 2 - 1;\n if (index < 0)\n return -1;\n if (index & 1) {\n ++index;\n break;\n }\n }\n }\n\n return -1;\n}\n\n\n//-----------------------------------------------------------------------------+\n// Combine two buddies into one node\nstatic void EMS_combine(struct emsMem *self, int64_t index) {\n for (; ;) {\n int64_t buddy = index - 1 + (index & 1) * 2;\n if (buddy < 0 || self->tree[buddy] != BUDDY_UNUSED) {\n self->tree[index] = BUDDY_UNUSED;\n while (((index = (index + 1) / 2 - 1) >= 0) && self->tree[index] == BUDDY_FULL) {\n self->tree[index] = BUDDY_SPLIT;\n }\n return;\n }\n index = (index + 1) / 2 - 1;\n }\n}\n\n\n//-----------------------------------------------------------------------------+\n// Release EMS memory back to the heap for reuse\nvoid emsMem_free(struct emsMem *self, size_t offset) {\n offset /= EMS_MEM_BLOCKSZ;\n assert(offset < (1UL << self->level));\n size_t left = 0;\n size_t length = 1UL << self->level;\n int64_t index = 0;\n\n for (; ;) {\n switch (self->tree[index]) {\n case BUDDY_USED:\n assert(offset == left);\n EMS_combine(self, index);\n return;\n case BUDDY_UNUSED:\n assert(0);\n return;\n default:\n length /= 2;\n if (offset < left + length) {\n index = index * 2 + 1;\n } else {\n left += length;\n index = index * 2 + 2;\n }\n break;\n }\n }\n}\n\n\n//-----------------------------------------------------------------------------+\n// Return the size of a block of memory\nsize_t emsMem_size(struct emsMem *self, size_t offset) {\n assert(offset < (1UL << self->level));\n size_t left = 0;\n size_t length = 1UL << self->level;\n size_t index = 0;\n\n for (; ;) {\n switch (self->tree[index]) {\n case BUDDY_USED:\n assert(offset == left);\n return length * EMS_MEM_BLOCKSZ;\n case BUDDY_UNUSED:\n assert(0);\n return length * EMS_MEM_BLOCKSZ;\n default:\n length /= 2;\n if (offset < left + length) {\n index = index * 2 + 1;\n } else {\n left += length;\n index = index * 2 + 2;\n }\n break;\n }\n }\n}\n\n\n//-----------------------------------------------------------------------------+\n// Diagnostic state dump\nstatic void EMS_dump(struct emsMem *self, size_t index, int32_t level) {\n switch (self->tree[index]) {\n case BUDDY_UNUSED:\n printf(\"(%lld:%ld)\", (long long int) EMS_index_offset(index, level, self->level),\n 1UL << (self->level - level));\n break;\n case BUDDY_USED:\n printf(\"[%lld:%ld]\", (long long int) EMS_index_offset(index, level, self->level),\n 1UL << (self->level - level));\n break;\n case BUDDY_FULL:\n printf(\"{\");\n EMS_dump(self, index * 2 + 1, level + 1);\n EMS_dump(self, index * 2 + 2, level + 1);\n printf(\"}\");\n break;\n default:\n printf(\"(\");\n EMS_dump(self, index * 2 + 1, level + 1);\n EMS_dump(self, index * 2 + 2, level + 1);\n printf(\")\");\n break;\n }\n}\n\nvoid emsMem_dump(struct emsMem *self) {\n EMS_dump(self, 0, 0);\n printf(\"\\n\");\n}\n"
},
{
"path": "src/ems_alloc.h",
"content": "/*-----------------------------------------------------------------------------+\n | Extended Memory Semantics (EMS) Version 1.4.0 |\n | Synthetic Semantics http://www.synsem.com/ mogill@synsem.com |\n +-----------------------------------------------------------------------------+\n | Copyright (c) 2011-2014, Synthetic Semantics LLC. All rights reserved. |\n | Copyright (c) 2015-2016, Jace A Mogill. All rights reserved. |\n | |\n | Redistribution and use in source and binary forms, with or without |\n | modification, are permitted provided that the following conditions are met: |\n | * Redistributions of source code must retain the above copyright |\n | notice, this list of conditions and the following disclaimer. |\n | * Redistributions in binary form must reproduce the above copyright |\n | notice, this list of conditions and the following disclaimer in the |\n | documentation and/or other materials provided with the distribution. |\n | * Neither the name of the Synthetic Semantics nor the names of its |\n | contributors may be used to endorse or promote products derived |\n | from this software without specific prior written permission. |\n | |\n | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS |\n | \"AS IS\" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT |\n | LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR |\n | A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL SYNTHETIC |\n | SEMANTICS LLC BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, |\n | EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, |\n | PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR |\n | PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF |\n | LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING |\n | NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS |\n | SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. |\n | |\n +-----------------------------------------------------------------------------*/\n#ifndef EMS_MEMORY_ALLOCATION_H\n#define EMS_MEMORY_ALLOCATION_H\n#include \n#include \n\n// The block size used by the memory allocator for allocating heap space.\n// May be any positive non-zero value\n#define EMS_MEM_BLOCKSZ 32\n\n\n// Buddy allocator control structure\nstruct emsMem {\n int32_t level;\n uint8_t tree[1];\n};\n\n\nstruct emsMem *emsMem_new(int level);\nvoid emsMem_delete(struct emsMem *);\nsize_t emsMem_alloc(struct emsMem *, size_t bytesRequested);\nvoid emsMem_free(struct emsMem *, size_t offset);\nsize_t emsMem_size(struct emsMem *, size_t offset);\nvoid emsMem_dump(struct emsMem *);\nsize_t emsNextPow2(int64_t x);\n\n#endif\n"
},
{
"path": "src/ems_proto.h",
"content": "/*-----------------------------------------------------------------------------+\n | Extended Memory Semantics (EMS) Version 1.5.0 |\n | Synthetic Semantics http://www.synsem.com/ mogill@synsem.com |\n +-----------------------------------------------------------------------------+\n | Copyright (c) 2011-2014, Synthetic Semantics LLC. All rights reserved. |\n | Copyright (c) 2015-2017, Jace A Mogill. All rights reserved. |\n | |\n | Redistribution and use in source and binary forms, with or without |\n | modification, are permitted provided that the following conditions are met: |\n | * Redistributions of source code must retain the above copyright |\n | notice, this list of conditions and the following disclaimer. |\n | * Redistributions in binary form must reproduce the above copyright |\n | notice, this list of conditions and the following disclaimer in the |\n | documentation and/or other materials provided with the distribution. |\n | * Neither the name of the Synthetic Semantics nor the names of its |\n | contributors may be used to endorse or promote products derived |\n | from this software without specific prior written permission. |\n | |\n | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS |\n | \"AS IS\" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT |\n | LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR |\n | A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL SYNTHETIC |\n | SEMANTICS LLC BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, |\n | EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, |\n | PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR |\n | PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF |\n | LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING |\n | NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS |\n | SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. |\n | |\n +-----------------------------------------------------------------------------*/\n#include \"ems_types.h\"\n\n// ---------------------------------------------------------------------------------\n// Non-exposed API functions\nint64_t EMSwriteIndexMap(const int mmapID, EMSvalueType *key);\nint64_t EMSkey2index(void *emsBuf, EMSvalueType *key, bool is_mapped);\nint64_t EMShashString(const char *key);\n\n\n// ---------------------------------------------------------------------------------\n// External API functions\nextern \"C\" int EMScriticalEnter(int mmapID, int timeout);\nextern \"C\" bool EMScriticalExit(int mmapID);\nextern \"C\" int EMSbarrier(int mmapID, int timeout);\nextern \"C\" bool EMSsingleTask(int mmapID);\nextern \"C\" bool EMScas(int mmapID, EMSvalueType *key,\n EMSvalueType *oldValue, EMSvalueType *newValue,\n EMSvalueType *returnValue);\nextern \"C\" bool EMSfaa(int mmapID, EMSvalueType *key, EMSvalueType *value, EMSvalueType *returnValue);\nextern \"C\" int EMSpush(int mmapID, EMSvalueType *value);\nextern \"C\" bool EMSpop(int mmapID, EMSvalueType *returnValue);\nextern \"C\" int EMSenqueue(int mmapID, EMSvalueType *value);\nextern \"C\" bool EMSdequeue(int mmapID, EMSvalueType *returnValue);\nextern \"C\" bool EMSloopInit(int mmapID, int32_t start, int32_t end, int32_t minChunk, int schedule_mode);\nextern \"C\" bool EMSloopChunk(int mmapID, int32_t *start, int32_t *end);\nextern \"C\" unsigned char EMStransitionFEtag(EMStag_t volatile *tag, EMStag_t volatile *mapTag, unsigned char oldFE, unsigned char newFE, unsigned char oldType);\nextern \"C\" bool EMSreadRW(const int mmapID, EMSvalueType *key, EMSvalueType *returnValue);\nextern \"C\" bool EMSreadFF(const int mmapID, EMSvalueType *key, EMSvalueType *returnValue);\nextern \"C\" bool EMSreadFE(const int mmapID, EMSvalueType *key, EMSvalueType *returnValue);\nextern \"C\" bool EMSread(const int mmapID, EMSvalueType *key, EMSvalueType *returnValue);\nextern \"C\" int EMSreleaseRW(const int mmapID, EMSvalueType *key);\nextern \"C\" bool EMSwriteXF(int mmapID, EMSvalueType *key, EMSvalueType *value);\nextern \"C\" bool EMSwriteXE(int mmapID, EMSvalueType *key, EMSvalueType *value);\nextern \"C\" bool EMSwriteEF(int mmapID, EMSvalueType *key, EMSvalueType *value);\nextern \"C\" bool EMSwrite(int mmapID, EMSvalueType *key, EMSvalueType *value);\nextern \"C\" bool EMSsetTag(int mmapID, EMSvalueType *key, bool is_full);\nextern \"C\" bool EMSdestroy(int mmapID, bool do_unlink);\nextern \"C\" bool EMSindex2key(int mmapID, int64_t idx, EMSvalueType *key);\nextern \"C\" bool EMSsync(int mmapID);\nextern \"C\" int EMSinitialize(int64_t nElements, // 0\n size_t heapSize, // 1\n bool useMap, // 2\n const char *filename, // 3\n bool persist, // 4\n bool useExisting, // 5\n bool doDataFill, // 6\n bool fillIsJSON, // 7\n EMSvalueType *fillValue, // 8\n bool doSetFEtags, // 9\n bool setFEtagsFull, // 10\n int EMSmyID, // 11\n bool pinThreads, // 12\n int32_t nThreads, // 13\n int32_t pctMLock ); // 14\n"
},
{
"path": "src/ems_types.h",
"content": "/*-----------------------------------------------------------------------------+\n | Extended Memory Semantics (EMS) Version 1.5.0 |\n | Synthetic Semantics http://www.synsem.com/ mogill@synsem.com |\n +-----------------------------------------------------------------------------+\n | Copyright (c) 2017, Jace A Mogill. All rights reserved. |\n | |\n | Redistribution and use in source and binary forms, with or without |\n | modification, are permitted provided that the following conditions are met: |\n | * Redistributions of source code must retain the above copyright |\n | notice, this list of conditions and the following disclaimer. |\n | * Redistributions in binary form must reproduce the above copyright |\n | notice, this list of conditions and the following disclaimer in the |\n | documentation and/or other materials provided with the distribution. |\n | * Neither the name of the Synthetic Semantics nor the names of its |\n | contributors may be used to endorse or promote products derived |\n | from this software without specific prior written permission. |\n | |\n | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS |\n | \"AS IS\" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT |\n | LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR |\n | A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL SYNTHETIC |\n | SEMANTICS LLC BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, |\n | EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, |\n | PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR |\n | PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF |\n | LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING |\n | NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS |\n | SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. |\n | |\n +-----------------------------------------------------------------------------*/\n#ifndef EMSPROJ_EMS_TYPES_H\n#define EMSPROJ_EMS_TYPES_H\n// Bitfields of a Tag Byte\n#define EMS_TYPE_NBITS_FE 2\n#define EMS_TYPE_NBITS_TYPE 3\n#define EMS_TYPE_NBITS_RW 3\n#define EMS_RW_NREADERS_MAX ((1 << EMS_TYPE_NBITS_RW) - 1)\ntypedef union {\n struct {\n unsigned char fe : EMS_TYPE_NBITS_FE;\n unsigned char type : EMS_TYPE_NBITS_TYPE;\n unsigned char rw : EMS_TYPE_NBITS_RW;\n } tags;\n unsigned char byte;\n} EMStag_t;\n\n#define EMS_VALUE_TYPE_INITIALIZER {.length=0, .value=NULL, .type=EMS_TYPE_INVALID}\n\n\n// Type-punning is now a warning in GCC, but this syntax is still okay\ntypedef union {\n double d;\n uint64_t u64;\n} EMSulong_double;\n\n\n// Internal EMS representation of a JSON value\ntypedef struct {\n size_t length; // Defined only for JSON and strings\n void *value;\n unsigned char type;\n} EMSvalueType;\n\n\n#endif\n//EMSPROJ_EMS_TYPES_H\n"
},
{
"path": "src/loops.cc",
"content": "/*-----------------------------------------------------------------------------+\n | Extended Memory Semantics (EMS) Version 1.3.0 |\n | Synthetic Semantics http://www.synsem.com/ mogill@synsem.com |\n +-----------------------------------------------------------------------------+\n | Copyright (c) 2011-2014, Synthetic Semantics LLC. All rights reserved. |\n | Copyright (c) 2015-2016, Jace A Mogill. All rights reserved. |\n | |\n | Redistribution and use in source and binary forms, with or without |\n | modification, are permitted provided that the following conditions are met: |\n | * Redistributions of source code must retain the above copyright |\n | notice, this list of conditions and the following disclaimer. |\n | * Redistributions in binary form must reproduce the above copyright |\n | notice, this list of conditions and the following disclaimer in the |\n | documentation and/or other materials provided with the distribution. |\n | * Neither the name of the Synthetic Semantics nor the names of its |\n | contributors may be used to endorse or promote products derived |\n | from this software without specific prior written permission. |\n | |\n | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS |\n | \"AS IS\" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT |\n | LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR |\n | A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL SYNTHETIC |\n | SEMANTICS LLC BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, |\n | EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, |\n | PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR |\n | PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF |\n | LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING |\n | NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS |\n | SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. |\n | |\n +-----------------------------------------------------------------------------*/\n#include \"ems.h\"\n\n\n//==================================================================\n// Parallel Loop -- context initialization\n//\nbool EMSloopInit(int mmapID, int32_t start, int32_t end, int32_t minChunk, int schedule_mode) {\n void *emsBuf = emsBufs[mmapID];\n int32_t *bufInt32 = (int32_t *) emsBuf;\n bool success = true;\n\n bufInt32[EMS_LOOP_IDX] = start;\n bufInt32[EMS_LOOP_START] = start;\n bufInt32[EMS_LOOP_END] = end;\n switch (schedule_mode) {\n case EMS_SCHED_GUIDED:\n bufInt32[EMS_LOOP_CHUNKSZ] = ((end - start) / 2) / bufInt32[EMS_CB_NTHREADS];\n if (bufInt32[EMS_LOOP_CHUNKSZ] < minChunk) bufInt32[EMS_LOOP_CHUNKSZ] = minChunk;\n bufInt32[EMS_LOOP_MINCHUNK] = minChunk;\n bufInt32[EMS_LOOP_SCHED] = EMS_SCHED_GUIDED;\n break;\n case EMS_SCHED_DYNAMIC:\n bufInt32[EMS_LOOP_CHUNKSZ] = 1;\n bufInt32[EMS_LOOP_MINCHUNK] = 1;\n bufInt32[EMS_LOOP_SCHED] = EMS_SCHED_DYNAMIC;\n break;\n default:\n fprintf(stderr, \"NodeJSloopInit: Unknown schedule modes\\n\");\n success = false;\n }\n return success;\n}\n\n\n//==================================================================\n// Determine the current block of iterations to assign to an\n// an idle thread\n// JQM TODO BUG -- convert to 64 bit using fe tags\n//\nbool EMSloopChunk(int mmapID, int32_t *start, int32_t *end) {\n void *emsBuf = emsBufs[mmapID];\n int32_t *bufInt32 = (int32_t *) emsBuf;\n\n int chunkSize = bufInt32[EMS_LOOP_CHUNKSZ];\n *start = __sync_fetch_and_add(&(bufInt32[EMS_LOOP_IDX]), chunkSize);\n *end = *start + chunkSize;\n\n if (*start > bufInt32[EMS_LOOP_END]) *end = 0;\n if (*end > bufInt32[EMS_LOOP_END]) *end = bufInt32[EMS_LOOP_END];\n if (bufInt32[EMS_LOOP_SCHED] == EMS_SCHED_GUIDED) {\n // Compute the size of the chunk the next thread should use\n int newSz = (int) ((bufInt32[EMS_LOOP_END] - *start) / 2) / bufInt32[EMS_CB_NTHREADS];\n if (newSz < bufInt32[EMS_LOOP_MINCHUNK]) newSz = bufInt32[EMS_LOOP_MINCHUNK];\n bufInt32[EMS_LOOP_CHUNKSZ] = newSz;\n }\n\n return true;\n}\n\n"
},
{
"path": "src/primitives.cc",
"content": "/*-----------------------------------------------------------------------------+\n | Extended Memory Semantics (EMS) Version 1.3.0 |\n | Synthetic Semantics http://www.synsem.com/ mogill@synsem.com |\n +-----------------------------------------------------------------------------+\n | Copyright (c) 2011-2014, Synthetic Semantics LLC. All rights reserved. |\n | Copyright (c) 2015-2016, Jace A Mogill. All rights reserved. |\n | |\n | Redistribution and use in source and binary forms, with or without |\n | modification, are permitted provided that the following conditions are met: |\n | * Redistributions of source code must retain the above copyright |\n | notice, this list of conditions and the following disclaimer. |\n | * Redistributions in binary form must reproduce the above copyright |\n | notice, this list of conditions and the following disclaimer in the |\n | documentation and/or other materials provided with the distribution. |\n | * Neither the name of the Synthetic Semantics nor the names of its |\n | contributors may be used to endorse or promote products derived |\n | from this software without specific prior written permission. |\n | |\n | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS |\n | \"AS IS\" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT |\n | LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR |\n | A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL SYNTHETIC |\n | SEMANTICS LLC BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, |\n | EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, |\n | PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR |\n | PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF |\n | LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING |\n | NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS |\n | SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. |\n | |\n +-----------------------------------------------------------------------------*/\n#include \"ems.h\"\n\n\n//==================================================================\n// Push onto stack\nint EMSpush(int mmapID, EMSvalueType *value) { // TODO: Eventually promote return value to 64bit\n void *emsBuf = emsBufs[mmapID];\n int64_t *bufInt64 = (int64_t *) emsBuf;\n EMStag_t *bufTags = (EMStag_t *) emsBuf;\n char *bufChar = (char *) emsBuf;\n EMStag_t newTag;\n\n // Wait until the stack top is full, then mark it busy while updating the stack\n EMStransitionFEtag(&bufTags[EMScbTag(EMS_ARR_STACKTOP)], NULL, EMS_TAG_FULL, EMS_TAG_BUSY, EMS_TAG_ANY);\n int32_t idx = bufInt64[EMScbData(EMS_ARR_STACKTOP)]; // TODO BUG: Truncating the full 64b range\n bufInt64[EMScbData(EMS_ARR_STACKTOP)]++;\n if (idx == bufInt64[EMScbData(EMS_ARR_NELEM)] - 1) {\n fprintf(stderr, \"EMSpush: Ran out of stack entries\\n\");\n return -1;\n }\n\n // Wait until the target memory at the top of the stack is empty\n newTag.byte = EMStransitionFEtag(&bufTags[EMSdataTag(idx)], NULL, EMS_TAG_EMPTY, EMS_TAG_BUSY, EMS_TAG_ANY);\n newTag.tags.rw = 0;\n newTag.tags.type = value->type;\n newTag.tags.fe = EMS_TAG_FULL;\n\n // Write the value onto the stack\n switch (newTag.tags.type) {\n case EMS_TYPE_BOOLEAN:\n case EMS_TYPE_INTEGER:\n case EMS_TYPE_FLOAT:\n bufInt64[EMSdataData(idx)] = (int64_t) value->value;\n break;\n case EMS_TYPE_JSON:\n case EMS_TYPE_STRING: {\n int64_t textOffset;\n EMS_ALLOC(textOffset, strlen((const char *) value->value) + 1, bufChar, \"EMSpush: out of memory to store string\\n\", -1);\n bufInt64[EMSdataData(idx)] = textOffset;\n strcpy(EMSheapPtr(textOffset), (const char *) value->value);\n }\n break;\n case EMS_TYPE_UNDEFINED:\n bufInt64[EMSdataData(idx)] = 0xdeadbeef;\n break;\n default:\n fprintf(stderr, \"EMSpush: Unknown value type\\n\");\n return -1;\n }\n\n // Mark the data on the stack as FULL\n bufTags[EMSdataTag(idx)].byte = newTag.byte;\n\n // Push is complete, Mark the stack pointer as full\n bufTags[EMScbTag(EMS_ARR_STACKTOP)].tags.fe = EMS_TAG_FULL;\n\n return idx;\n}\n\n\n//==================================================================\n// Pop data from stack\n//\nbool EMSpop(int mmapID, EMSvalueType *returnValue) {\n void *emsBuf = emsBufs[mmapID];\n int64_t *bufInt64 = (int64_t *) emsBuf;\n EMStag_t *bufTags = (EMStag_t *) emsBuf;\n char *bufChar = (char *) emsBuf;\n EMStag_t dataTag;\n\n // Wait until the stack pointer is full and mark it empty while pop is performed\n EMStransitionFEtag(&bufTags[EMScbTag(EMS_ARR_STACKTOP)], NULL, EMS_TAG_FULL, EMS_TAG_BUSY, EMS_TAG_ANY);\n bufInt64[EMScbData(EMS_ARR_STACKTOP)]--;\n int64_t idx = bufInt64[EMScbData(EMS_ARR_STACKTOP)];\n if (idx < 0) {\n // Stack is empty, return undefined\n bufInt64[EMScbData(EMS_ARR_STACKTOP)] = 0;\n bufTags[EMScbTag(EMS_ARR_STACKTOP)].tags.fe = EMS_TAG_FULL;\n returnValue->type = EMS_TYPE_UNDEFINED;\n returnValue->value = (void *) 0xf00dd00f;\n return true;\n }\n // Wait until the data pointed to by the stack pointer is full, then mark it\n // busy while it is copied, and set it to EMPTY when finished\n dataTag.byte = EMStransitionFEtag(&bufTags[EMSdataTag(idx)], NULL, EMS_TAG_FULL, EMS_TAG_BUSY, EMS_TAG_ANY);\n returnValue->type = dataTag.tags.type;\n switch (dataTag.tags.type) {\n case EMS_TYPE_BOOLEAN:\n case EMS_TYPE_INTEGER:\n case EMS_TYPE_FLOAT: {\n returnValue->value = (void *) bufInt64[EMSdataData(idx)];\n bufTags[EMSdataTag(idx)].tags.fe = EMS_TAG_EMPTY;\n bufTags[EMScbTag(EMS_ARR_STACKTOP)].tags.fe = EMS_TAG_FULL;\n return true;\n }\n case EMS_TYPE_JSON:\n case EMS_TYPE_STRING: {\n size_t memStrLen = strlen(EMSheapPtr(bufInt64[EMSdataData(idx)])); // TODO: Use size of allocation, not strlen\n returnValue->value = malloc(memStrLen + 1); // freed in NodeJSfaa\n if(returnValue->value == NULL) {\n fprintf(stderr, \"EMSpop: Unable to allocate space to return stack top string\\n\");\n return false;\n }\n strcpy((char *) returnValue->value, EMSheapPtr(bufInt64[EMSdataData(idx)]));\n EMS_FREE(bufInt64[EMSdataData(idx)]);\n bufTags[EMSdataTag(idx)].tags.fe = EMS_TAG_EMPTY;\n bufTags[EMScbTag(EMS_ARR_STACKTOP)].tags.fe = EMS_TAG_FULL;\n return true;\n }\n case EMS_TYPE_UNDEFINED: {\n bufTags[EMSdataTag(idx)].tags.fe = EMS_TAG_EMPTY;\n bufTags[EMScbTag(EMS_ARR_STACKTOP)].tags.fe = EMS_TAG_FULL;\n returnValue->value = (void *) 0xdeadbeef;\n return true;\n }\n default:\n fprintf(stderr, \"EMSpop: ERROR - unknown top of stack data type\\n\");\n return false;\n }\n}\n\n\n//==================================================================\n// Enqueue data\n// Heap top and bottom are monotonically increasing, but the index\n// returned is a circular buffer.\nint EMSenqueue(int mmapID, EMSvalueType *value) { // TODO: Eventually promote return value to 64bit\n void *emsBuf = emsBufs[mmapID];\n int64_t *bufInt64 = (int64_t *) emsBuf;\n EMStag_t *bufTags = (EMStag_t *) emsBuf;\n char *bufChar = (char *) emsBuf;\n\n // Wait until the heap top is full, and mark it busy while data is enqueued\n EMStransitionFEtag(&bufTags[EMScbTag(EMS_ARR_STACKTOP)], NULL, EMS_TAG_FULL, EMS_TAG_BUSY, EMS_TAG_ANY);\n int32_t idx = bufInt64[EMScbData(EMS_ARR_STACKTOP)] % bufInt64[EMScbData(EMS_ARR_NELEM)]; // TODO: BUG This could be truncated\n bufInt64[EMScbData(EMS_ARR_STACKTOP)]++;\n if (bufInt64[EMScbData(EMS_ARR_STACKTOP)] - bufInt64[EMScbData(EMS_ARR_Q_BOTTOM)] >\n bufInt64[EMScbData(EMS_ARR_NELEM)]) {\n fprintf(stderr, \"EMSenqueue: Ran out of stack entries\\n\");\n return -1;\n }\n\n // Wait for data pointed to by heap top to be empty, then set to Full while it is filled\n bufTags[EMSdataTag(idx)].tags.rw = 0;\n bufTags[EMSdataTag(idx)].tags.type = value->type;\n switch (bufTags[EMSdataTag(idx)].tags.type) {\n case EMS_TYPE_BOOLEAN:\n case EMS_TYPE_INTEGER:\n case EMS_TYPE_FLOAT:\n bufInt64[EMSdataData(idx)] = (int64_t) value->value;\n break;\n case EMS_TYPE_JSON:\n case EMS_TYPE_STRING: {\n int64_t textOffset;\n EMS_ALLOC(textOffset, strlen((const char *) value->value) + 1, bufChar, \"EMSenqueue: out of memory to store string\\n\", -1);\n bufInt64[EMSdataData(idx)] = textOffset;\n strcpy(EMSheapPtr(textOffset), (const char *) value->value);\n }\n break;\n case EMS_TYPE_UNDEFINED:\n bufInt64[EMSdataData(idx)] = 0xdeadbeef;\n break;\n default:\n fprintf(stderr, \"EMSenqueue: Unknown value type\\n\");\n return -1;\n }\n\n // Set the tag on the data to FULL\n bufTags[EMSdataTag(idx)].tags.fe = EMS_TAG_FULL;\n\n // Enqueue is complete, set the tag on the heap to to FULL\n bufTags[EMScbTag(EMS_ARR_STACKTOP)].tags.fe = EMS_TAG_FULL;\n return idx;\n}\n\n\n//==================================================================\n// Dequeue\nbool EMSdequeue(int mmapID, EMSvalueType *returnValue) {\n void *emsBuf = emsBufs[mmapID];\n int64_t *bufInt64 = (int64_t *) emsBuf;\n EMStag_t *bufTags = (EMStag_t *) emsBuf;\n char *bufChar = (char *) emsBuf;\n EMStag_t dataTag;\n\n // Wait for bottom of heap pointer to be full, and mark it busy while data is dequeued\n EMStransitionFEtag(&bufTags[EMScbTag(EMS_ARR_Q_BOTTOM)], NULL, EMS_TAG_FULL, EMS_TAG_BUSY, EMS_TAG_ANY);\n int64_t idx = bufInt64[EMScbData(EMS_ARR_Q_BOTTOM)] % bufInt64[EMScbData(EMS_ARR_NELEM)];\n // If Queue is empty, return undefined\n if (bufInt64[EMScbData(EMS_ARR_Q_BOTTOM)] >= bufInt64[EMScbData(EMS_ARR_STACKTOP)]) {\n bufInt64[EMScbData(EMS_ARR_Q_BOTTOM)] = bufInt64[EMScbData(EMS_ARR_STACKTOP)];\n bufTags[EMScbTag(EMS_ARR_Q_BOTTOM)].tags.fe = EMS_TAG_FULL;\n returnValue->type = EMS_TYPE_UNDEFINED;\n returnValue->value = (void *) 0xf00dd00f;\n return true;\n }\n\n bufInt64[EMScbData(EMS_ARR_Q_BOTTOM)]++;\n // Wait for the data pointed to by the bottom of the heap to be full,\n // then mark busy while copying it, and finally set it to empty when done\n dataTag.byte = EMStransitionFEtag(&bufTags[EMSdataTag(idx)], NULL, EMS_TAG_FULL, EMS_TAG_BUSY, EMS_TAG_ANY);\n dataTag.tags.fe = EMS_TAG_EMPTY;\n returnValue->type = dataTag.tags.type;\n switch (dataTag.tags.type) {\n case EMS_TYPE_BOOLEAN:\n case EMS_TYPE_INTEGER:\n case EMS_TYPE_FLOAT: {\n returnValue->value = (void *) bufInt64[EMSdataData(idx)];\n bufTags[EMSdataTag(idx)].byte = dataTag.byte;\n bufTags[EMScbTag(EMS_ARR_Q_BOTTOM)].tags.fe = EMS_TAG_FULL;\n return true;\n }\n case EMS_TYPE_JSON:\n case EMS_TYPE_STRING: {\n bufTags[EMSdataTag(idx)].byte = dataTag.byte;\n bufTags[EMScbTag(EMS_ARR_Q_BOTTOM)].tags.fe = EMS_TAG_FULL;\n size_t memStrLen = strlen(EMSheapPtr(bufInt64[EMSdataData(idx)])); // TODO: Use size of allocation, not strlen\n returnValue->value = malloc(memStrLen + 1); // freed in NodeJSfaa\n if(returnValue->value == NULL) {\n fprintf(stderr, \"EMSdequeue: Unable to allocate space to return queue head string\\n\");\n return false;\n }\n strcpy((char *) returnValue->value, EMSheapPtr(bufInt64[EMSdataData(idx)]));\n EMS_FREE(bufInt64[EMSdataData(idx)]);\n return true;\n }\n case EMS_TYPE_UNDEFINED: {\n bufTags[EMSdataTag(idx)].byte = dataTag.byte;\n bufTags[EMScbTag(EMS_ARR_Q_BOTTOM)].tags.fe = EMS_TAG_FULL;\n returnValue->value = (void *) 0xdeadbeef;\n return true;\n }\n default:\n fprintf(stderr, \"EMSdequeue: ERROR - unknown type at head of queue\\n\");\n return false;\n }\n}\n"
},
{
"path": "src/rmw.cc",
"content": "/*-----------------------------------------------------------------------------+\n | Extended Memory Semantics (EMS) Version 1.3.0 |\n | Synthetic Semantics http://www.synsem.com/ mogill@synsem.com |\n +-----------------------------------------------------------------------------+\n | Copyright (c) 2011-2014, Synthetic Semantics LLC. All rights reserved. |\n | Copyright (c) 2015-2016, Jace A Mogill. All rights reserved. |\n | |\n | Redistribution and use in source and binary forms, with or without |\n | modification, are permitted provided that the following conditions are met: |\n | * Redistributions of source code must retain the above copyright |\n | notice, this list of conditions and the following disclaimer. |\n | * Redistributions in binary form must reproduce the above copyright |\n | notice, this list of conditions and the following disclaimer in the |\n | documentation and/or other materials provided with the distribution. |\n | * Neither the name of the Synthetic Semantics nor the names of its |\n | contributors may be used to endorse or promote products derived |\n | from this software without specific prior written permission. |\n | |\n | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS |\n | \"AS IS\" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT |\n | LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR |\n | A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL SYNTHETIC |\n | SEMANTICS LLC BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, |\n | EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, |\n | PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR |\n | PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF |\n | LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING |\n | NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS |\n | SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. |\n | |\n +-----------------------------------------------------------------------------*/\n#include \"ems.h\"\n\n//==================================================================\n// Fetch and Add Atomic Memory Operation\n// Returns a+b where a is data in EMS memory and b is an argument\nbool EMSfaa(int mmapID, EMSvalueType *key, EMSvalueType *value, EMSvalueType *returnValue) {\n void *emsBuf = emsBufs[mmapID];\n volatile EMStag_t *bufTags = (EMStag_t *) emsBuf;\n int64_t idx = EMSwriteIndexMap(mmapID, key);\n volatile int64_t *bufInt64 = (int64_t *) emsBuf;\n volatile double *bufDouble = (double *) emsBuf;\n char *bufChar = (char *) emsBuf;\n EMStag_t oldTag;\n\n if (idx < 0 || idx >= bufInt64[EMScbData(EMS_ARR_NELEM)]) {\n fprintf(stderr, \"EMSfaa: index out of bounds\\n\");\n return false;\n }\n\n volatile EMStag_t *maptag;\n if (EMSisMapped) { maptag = &bufTags[EMSmapTag(idx)]; }\n else { maptag = NULL; }\n // Wait until the data is FULL, mark it busy while FAA is performed\n oldTag.byte = EMStransitionFEtag(&bufTags[EMSdataTag(idx)], maptag,\n EMS_TAG_FULL, EMS_TAG_BUSY, EMS_TAG_ANY);\n\n oldTag.tags.fe = EMS_TAG_FULL; // When written back, mark FULL\n switch (oldTag.tags.type) {\n case EMS_TYPE_BOOLEAN: { // Bool + _______\n bool retBool = bufInt64[EMSdataData(idx)]; // Read original value in memory\n returnValue->value = (void *) retBool;\n returnValue->type = EMS_TYPE_BOOLEAN;\n switch (value->type) {\n case EMS_TYPE_INTEGER: // Bool + Int\n bufInt64[EMSdataData(idx)] += (int64_t) value->value;\n oldTag.tags.type = EMS_TYPE_INTEGER;\n break;\n case EMS_TYPE_FLOAT: { // Bool + Float\n EMSulong_double alias;\n alias.u64 = (uint64_t) value->value;\n bufDouble[EMSdataData(idx)] =\n (double) bufInt64[EMSdataData(idx)] + alias.d;\n oldTag.tags.type = EMS_TYPE_FLOAT;\n }\n break;\n case EMS_TYPE_UNDEFINED: // Bool + undefined\n bufDouble[EMSdataData(idx)] = NAN;\n oldTag.tags.type = EMS_TYPE_FLOAT;\n break;\n case EMS_TYPE_BOOLEAN: // Bool + Bool\n bufInt64[EMSdataData(idx)] += (int64_t) value->value;\n oldTag.tags.type = EMS_TYPE_INTEGER;\n break;\n case EMS_TYPE_STRING: { // Bool + string\n int64_t textOffset;\n EMS_ALLOC(textOffset, value->length + 1 + 5, // String length + Terminating null + 'false'\n bufChar, \"EMSfaa(bool+string): out of memory to store string\\n\", false);\n sprintf(EMSheapPtr(textOffset), \"%s%s\",\n bufInt64[EMSdataData(idx)] ? \"true\" : \"false\", (const char *) value->value);\n bufInt64[EMSdataData(idx)] = textOffset;\n oldTag.tags.type = EMS_TYPE_STRING;\n }\n break;\n default:\n fprintf(stderr, \"EMSfaa: Data is BOOL, but FAA arg type is unknown\\n\");\n return false;\n\n }\n // Write the new type and set the tag to Full, then return the original value\n bufTags[EMSdataTag(idx)].byte = oldTag.byte;\n if (EMSisMapped) bufTags[EMSmapTag(idx)].tags.fe = EMS_TAG_FULL;\n return true;\n } // End of: Bool + ___\n\n case EMS_TYPE_INTEGER: {\n int64_t retInt = bufInt64[EMSdataData(idx)]; // Read original value in memory\n returnValue->type = EMS_TYPE_INTEGER;\n returnValue->value = (void *) retInt;\n switch (value->type) {\n case EMS_TYPE_INTEGER: { // Int + int\n int64_t memInt = bufInt64[EMSdataData(idx)] + (int64_t) value->value;\n // TODO: Magic max int promotion to float\n if (memInt >= (1 << 30)) { // Possible integer overflow, convert to float\n bufDouble[EMSdataData(idx)] = (double) memInt;\n oldTag.tags.type = EMS_TYPE_FLOAT;\n } else { // Did not overflow to flow, still an integer\n bufInt64[EMSdataData(idx)] = memInt;\n }\n }\n break;\n case EMS_TYPE_FLOAT: { // Int + float\n EMSulong_double alias;\n alias.u64 = (uint64_t) value->value;\n bufDouble[EMSdataData(idx)] =\n (double) bufInt64[EMSdataData(idx)] + alias.d;\n oldTag.tags.type = EMS_TYPE_FLOAT;\n }\n break;\n case EMS_TYPE_UNDEFINED: // Int + undefined\n bufDouble[EMSdataData(idx)] = NAN;\n oldTag.tags.type = EMS_TYPE_FLOAT;\n break;\n case EMS_TYPE_BOOLEAN: // Int + bool\n bufInt64[EMSdataData(idx)] += (int64_t) value->value;\n break;\n case EMS_TYPE_STRING: { // int + string\n int64_t textOffset;\n EMS_ALLOC(textOffset, value->length + 1 + MAX_NUMBER2STR_LEN,\n bufChar, \"EMSfaa(int+string): out of memory to store string\\n\", false);\n sprintf(EMSheapPtr(textOffset), \"%lld%s\",\n (long long int) bufInt64[EMSdataData(idx)], (const char *) value->value);\n bufInt64[EMSdataData(idx)] = textOffset;\n oldTag.tags.type = EMS_TYPE_STRING;\n }\n break;\n default:\n fprintf(stderr, \"EMSfaa: Data is INT, but FAA arg type is unknown\\n\");\n return false;\n }\n // Write the new type and set the tag to Full, then return the original value\n bufTags[EMSdataTag(idx)].byte = oldTag.byte;\n if (EMSisMapped) bufTags[EMSmapTag(idx)].tags.fe = EMS_TAG_FULL;\n return true;\n } // End of: Integer + ____\n\n case EMS_TYPE_FLOAT: {\n double retDbl = bufDouble[EMSdataData(idx)];\n returnValue->type = EMS_TYPE_FLOAT;\n EMSulong_double alias;\n alias.d = retDbl;\n returnValue->value = (void *) alias.u64;\n\n switch (value->type) {\n case EMS_TYPE_INTEGER: // Float + int\n bufDouble[EMSdataData(idx)] += (double) ((int64_t) value->value);\n break;\n case EMS_TYPE_FLOAT: { // Float + float\n EMSulong_double alias;\n alias.u64 = (uint64_t) value->value;\n bufDouble[EMSdataData(idx)] += alias.d;\n }\n break;\n case EMS_TYPE_BOOLEAN: // Float + boolean\n bufDouble[EMSdataData(idx)] += (double) ((int64_t) value->value);\n break;\n case EMS_TYPE_STRING: { // Float + string\n int64_t textOffset;\n EMS_ALLOC(textOffset, value->length + 1 + MAX_NUMBER2STR_LEN,\n bufChar, \"EMSfaa(float+string): out of memory to store string\\n\", false);\n sprintf(EMSheapPtr(textOffset), \"%lf%s\", bufDouble[EMSdataData(idx)], (const char *) value->value);\n bufInt64[EMSdataData(idx)] = textOffset;\n oldTag.tags.type = EMS_TYPE_STRING;\n }\n break;\n case EMS_TYPE_UNDEFINED: // Float + Undefined\n bufDouble[EMSdataData(idx)] = NAN;\n break;\n default:\n fprintf(stderr, \"EMSfaa: Data is FLOAT, but arg type unknown\\n\");\n return false;\n }\n // Write the new type and set the tag to Full, then return the original value\n bufTags[EMSdataTag(idx)].byte = oldTag.byte;\n if (EMSisMapped) bufTags[EMSmapTag(idx)].tags.fe = EMS_TAG_FULL;\n return true;\n } // End of: float + _______\n\n case EMS_TYPE_STRING: {\n // size_t oldStrLen = (size_t) emsMem_size(EMS_MEM_MALLOCBOT(bufChar), bufInt64[EMSdataData(idx)]);\n size_t oldStrLen = strlen(EMSheapPtr(bufInt64[EMSdataData(idx)]));\n returnValue->type = EMS_TYPE_STRING;\n returnValue->value = malloc(oldStrLen + 1); // freed in NodeJSfaa\n if(returnValue->value == NULL) {\n fprintf(stderr, \"EMSfaa: Unable to malloc temporary old string\\n\");\n return false;\n }\n strcpy((char *) returnValue->value, EMSheapPtr(bufInt64[EMSdataData(idx)]));\n int64_t textOffset;\n size_t len;\n switch (value->type) {\n case EMS_TYPE_INTEGER: // string + int\n len = oldStrLen + 1 + MAX_NUMBER2STR_LEN;\n EMS_ALLOC(textOffset, len, bufChar, \"EMSfaa(string+int): out of memory to store string\\n\", false);\n sprintf(EMSheapPtr(textOffset), \"%s%lld\",\n EMSheapPtr(bufInt64[EMSdataData(idx)]),\n (long long int) value->value);\n break;\n case EMS_TYPE_FLOAT: { // string + dbl\n EMSulong_double alias;\n alias.u64 = (uint64_t) value->value;\n len = oldStrLen + 1 + MAX_NUMBER2STR_LEN;\n EMS_ALLOC(textOffset, len, bufChar, \"EMSfaa(string+dbl): out of memory to store string\\n\", false);\n sprintf(EMSheapPtr(textOffset), \"%s%lf\",\n EMSheapPtr(bufInt64[EMSdataData(idx)]), alias.d);\n }\n break;\n case EMS_TYPE_STRING: { // string + string\n len = oldStrLen + 1 + value->length;\n EMS_ALLOC(textOffset, len, bufChar, \"EMSfaa(string+string): out of memory to store string\\n\", false);\n sprintf(EMSheapPtr(textOffset), \"%s%s\",\n EMSheapPtr(bufInt64[EMSdataData(idx)]), (const char *) value->value);\n }\n break;\n case EMS_TYPE_BOOLEAN: // string + bool\n len = strlen(EMSheapPtr(bufInt64[EMSdataData(idx)])) + 1 + 5; // 5==strlen(\"false\")\n EMS_ALLOC(textOffset, len, bufChar, \"EMSfaa(string+bool): out of memory to store string\\n\", false);\n sprintf(EMSheapPtr(textOffset), \"%s%s\",\n EMSheapPtr(bufInt64[EMSdataData(idx)]), (bool) value->value ? \"true\" : \"false\");\n break;\n case EMS_TYPE_UNDEFINED: // string + undefined\n len = strlen(EMSheapPtr(bufInt64[EMSdataData(idx)])) + 1 + 9; // 9 == strlen(\"undefined\");\n EMS_ALLOC(textOffset, len, bufChar, \"EMSfaa(string+undefined): out of memory to store string\\n\", false);\n sprintf(EMSheapPtr(textOffset), \"%s%s\",\n EMSheapPtr(bufInt64[EMSdataData(idx)]), \"undefined\");\n break;\n default:\n fprintf(stderr, \"EMSfaa(string+?): Unknown data type\\n\");\n return false;\n }\n EMS_FREE(bufInt64[EMSdataData(idx)]);\n bufInt64[EMSdataData(idx)] = textOffset;\n oldTag.tags.type = EMS_TYPE_STRING;\n // Write the new type and set the tag to Full, then return the original value\n bufTags[EMSdataTag(idx)].byte = oldTag.byte;\n if (EMSisMapped) bufTags[EMSmapTag(idx)].tags.fe = EMS_TAG_FULL;\n // return value was set at the top of this block\n return true;\n } // End of: String + __________\n\n case EMS_TYPE_UNDEFINED: {\n returnValue->type = EMS_TYPE_UNDEFINED;\n returnValue->value = (void *) 0xf00dd00f;\n switch (value->type) { // Undefined + Int, dloat, bool, or undef\n case EMS_TYPE_INTEGER:\n case EMS_TYPE_FLOAT:\n case EMS_TYPE_BOOLEAN:\n case EMS_TYPE_UNDEFINED:\n bufDouble[EMSdataData(idx)] = NAN;\n oldTag.tags.type = EMS_TYPE_FLOAT;\n break;\n case EMS_TYPE_STRING: { // Undefined + string\n int64_t textOffset;\n EMS_ALLOC(textOffset, value->length + 1 + 3, // 3 = strlen(\"NaN\");\n bufChar, \"EMSfaa(undef+String): out of memory to store string\\n\", false);\n sprintf(EMSheapPtr(textOffset), \"NaN%s\", (const char *) value->value);\n bufInt64[EMSdataData(idx)] = textOffset;\n oldTag.tags.type = EMS_TYPE_UNDEFINED;\n }\n break;\n default:\n fprintf(stderr, \"EMSfaa(Undefined+___: Unknown stored data type\\n\");\n return false;\n }\n // Write the new type and set the tag to Full, then return the original value\n bufTags[EMSdataTag(idx)].byte = oldTag.byte;\n if (EMSisMapped) bufTags[EMSmapTag(idx)].tags.fe = EMS_TAG_FULL;\n return true;\n }\n default:\n fprintf(stderr, \"EMSfaa(?+___: Unknown stored data type\\n\");\n return false;\n }\n // fprintf(stderr, \"EMSfaa: Unknown everything -- this statement is unreachable\\n\");\n return false;\n}\n\n\n//==================================================================\n// Atomic Compare and Swap\nbool EMScas(int mmapID, EMSvalueType *key,\n EMSvalueType *oldValue, EMSvalueType *newValue,\n EMSvalueType *returnValue) {\n void *emsBuf = emsBufs[mmapID];\n volatile int64_t *bufInt64 = (int64_t *) emsBuf;\n int64_t idx = EMSkey2index(emsBuf, key, EMSisMapped);\n char * bufChar = (char *) emsBuf;\n volatile EMStag_t *bufTags = (EMStag_t *) emsBuf;\n EMStag_t newTag;\n int64_t textOffset;\n int swapped = false;\n\n if ((!EMSisMapped && idx < 0) || idx >= bufInt64[EMScbData(EMS_ARR_NELEM)]) {\n fprintf(stderr, \"EMScas: index out of bounds\\n\");\n return false;\n }\n\n size_t memStrLen;\n unsigned char memType;\nretry_on_undefined:\n if(EMSisMapped && idx < 0) {\n memType = EMS_TYPE_UNDEFINED;\n } else {\n // Wait for the memory to be Full, then mark it Busy while CAS works\n volatile EMStag_t *maptag;\n if (EMSisMapped) { maptag = &bufTags[EMSmapTag(idx)]; }\n else { maptag = NULL; }\n // Wait until the data is FULL, mark it busy while FAA is performed\n EMStransitionFEtag(&bufTags[EMSdataTag(idx)], maptag,\n EMS_TAG_FULL, EMS_TAG_BUSY, EMS_TAG_ANY);\n memType = bufTags[EMSdataTag(idx)].tags.type;\n }\n\n // Read the value in memory\n returnValue->type = memType;\n switch (memType) {\n case EMS_TYPE_UNDEFINED:\n returnValue->value = (void *) 0xf00dcafe;\n break;\n case EMS_TYPE_BOOLEAN:\n case EMS_TYPE_INTEGER:\n case EMS_TYPE_FLOAT:\n returnValue->value = (void *) bufInt64[EMSdataData(idx)];\n break;\n case EMS_TYPE_JSON:\n case EMS_TYPE_STRING:\n memStrLen = strlen(EMSheapPtr(bufInt64[EMSdataData(idx)]));\n returnValue->value = malloc(memStrLen + 1); // freed in NodeJSfaa\n if(returnValue->value == NULL) {\n fprintf(stderr, \"EMScas: Unable to allocate space to return old string\\n\");\n return false;\n }\n strcpy((char *) returnValue->value, EMSheapPtr(bufInt64[EMSdataData(idx)]));\n break;\n default:\n fprintf(stderr, \"EMScas: memType not recognized\\n\");\n return false;\n }\n\n // Compare the value in memory the the \"old\" CAS value\n if (oldValue->type == memType) {\n // Allocate on Write: If this memory was undefined (ie: unallocated),\n // allocate the index map, store the undefined, and start over again.\n if(EMSisMapped && idx < 0) {\n idx = EMSwriteIndexMap(mmapID, key);\n if (idx < 0) {\n fprintf(stderr, \"EMScas: Not able to allocate map on CAS of undefined data\\n\");\n return false;\n }\n bufInt64[EMSdataData(idx)] = 0xcafebabe;\n newTag.tags.fe = EMS_TAG_FULL;\n newTag.tags.rw = 0;\n newTag.tags.type = EMS_TYPE_UNDEFINED;\n bufTags[EMSdataTag(idx)].byte = newTag.byte;\n goto retry_on_undefined;\n }\n switch (memType) {\n case EMS_TYPE_UNDEFINED:\n swapped = true;\n break;\n case EMS_TYPE_BOOLEAN:\n case EMS_TYPE_INTEGER:\n case EMS_TYPE_FLOAT:\n if (returnValue->value == oldValue->value)\n swapped = true;\n break;\n case EMS_TYPE_JSON:\n case EMS_TYPE_STRING:\n if (strcmp((const char *) returnValue->value, (const char *) oldValue->value) == 0) {\n swapped = true;\n }\n break;\n default:\n fprintf(stderr, \"EMScas: oldTag not recognized\");\n return false;\n }\n }\n\n // If memory==old then write the new value\n newTag.tags.fe = EMS_TAG_FULL;\n newTag.tags.rw = 0;\n newTag.tags.type = memType;\n if (swapped) {\n if (memType == EMS_TYPE_STRING || memType == EMS_TYPE_JSON)\n EMS_FREE((size_t) bufInt64[EMSdataData(idx)]);\n newTag.tags.type = newValue->type;\n switch (newValue->type) {\n case EMS_TYPE_UNDEFINED:\n bufInt64[EMSdataData(idx)] = 0xbeeff00d;\n break;\n case EMS_TYPE_BOOLEAN:\n case EMS_TYPE_INTEGER:\n case EMS_TYPE_FLOAT:\n bufInt64[EMSdataData(idx)] = (int64_t) newValue->value;\n break;\n case EMS_TYPE_JSON:\n case EMS_TYPE_STRING:\n EMS_ALLOC(textOffset, newValue->length + 1,\n bufChar, \"EMScas(string): out of memory to store string\\n\", false);\n strcpy(EMSheapPtr(textOffset), (const char *) newValue->value);\n bufInt64[EMSdataData(idx)] = textOffset;\n break;\n default:\n fprintf(stderr, \"EMScas(): Unrecognized new type\\n\");\n return false;\n }\n }\n\n // Set the tag back to Full and return the original value\n bufTags[EMSdataTag(idx)].byte = newTag.byte;\n // If there is a map, set the map's tag back to full\n if (EMSisMapped)\n bufTags[EMSmapTag(idx)].tags.fe = EMS_TAG_FULL;\n\n return true;\n}\n"
}
]
![](\"./fsmSimple.svg\")
\n \n\t![](\"./memLayoutLogical.svg\")
\n \n\t![](\"./nodejs.svg\")
\n fork primitives, \n creates the additional threads executing \n using one of two execution models: fork-join or Bulk Synchronous Parallel (BSP).\n BSP invokes the same script as the master thread (found in ![](\"./BSPvsForkJoin.svg\")
\n ![](\"./blockDiagram.svg\")
\n ![](\"tm_from_q.svg\")
\n ![](\"./TMfromLoop.png\")
\n ![](\"./ccab.png\")
\n ![](\"./wordcount.svg\")
\n ![](\"./typesOfParallelism.svg\")
\n\t ![](\"./timelines.svg\")
\n\t ![](\"./strong_scaling.svg\")
\n ![](\"./weak_scaling.svg\")
\n ![](\"./Flag_of_Cascadia.svg\")
\n ![](\"./synsem_logo_black.svg\"){kind=logo}
\n\t![](\"../../Docs/ems_js_py.gif\")
\n"
},
{
"path": "Examples/Interlanguage/interlanguage.js",
"content": "/*-----------------------------------------------------------------------------+\n | Extended Memory Semantics (EMS) Version 1.4.5 |\n | Synthetic Semantics http://www.synsem.com/ mogill@synsem.com |\n +-----------------------------------------------------------------------------+\n | Copyright (c) 2017, Jace A Mogill. All rights reserved. |\n | |\n | Redistribution and use in source and binary forms, with or without |\n | modification, are permitted provided that the following conditions are met: |\n | * Redistributions of source code must retain the above copyright |\n | notice, this list of conditions and the following disclaimer. |\n | * Redistributions in binary form must reproduce the above copyright |\n | notice, this list of conditions and the following disclaimer in the |\n | documentation and/or other materials provided with the distribution. |\n | * Neither the name of the Synthetic Semantics nor the names of its |\n | contributors may be used to endorse or promote products derived |\n | from this software without specific prior written permission. |\n | |\n | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS |\n | \"AS IS\" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT |\n | LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR |\n | A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL SYNTHETIC |\n | SEMANTICS LLC BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, |\n | EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, |\n | PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR |\n | PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF |\n | LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING |\n | NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS |\n | SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. |\n | |\n +-----------------------------------------------------------------------------*/\n/*\n Start this JS program first, then start the Python program\n\n Possible Orders for Starting Processes\n ---------------------------------------------\n\n A persistent EMS file already exists\n 1. A one-time initialization process creates the EMS array\n 2. Any number of processes my attach to the EMS array in\n any order.\n\n\n A new EMS array will be created\n 1. The first program to start must create the EMS file with\n the \"useExisting : false\" attribute\n 2. Subsequent programs attach to the new EMS file with the\n \"useExisting : true\" attribute\n\n */\n\"use strict\";\n// Initialize EMS with 1 process, no thread-CPU affinity,\n// \"user\" mode parallelism, and a unique namespace for EMS runtime\n// (\"jsExample\") to keep the JS program distinct from Python EMS\n// program also running.\nlet ems = require(\"ems\")(1, false, \"user\", \"jsExample\");\n\nif (Number(process.version[1]) < 6) {\n console.log('Requires Node.js version 6.x or higher -- will not work with polyfill Harmony');\n process.exit(1);\n}\n\n\nconst maxNKeys = 100;\nconst bytesPerEntry = 100; // Bytes of storage per key, used for key (dictionary word) itself\nlet shared = ems.new({\n \"ES6proxies\": true, // Enable native JS object-like syntax\n \"useExisting\": false, // Create a new EMS memory area, do not use existing one if present\n \"filename\": \"/tmp/interlanguage.ems\", // Persistent EMS array's filename\n \"dimensions\": maxNKeys, // Maximum # of different keys the array can store\n \"heapSize\": maxNKeys * bytesPerEntry,\n \"setFEtags\": \"empty\", // Set default full/empty state of EMS memory to empty\n \"doSetFEtags\": true,\n \"useMap\": true // Use a key-index mapping, not integer indexes\n});\n\n\n\n//------------------------------------------------------------------------------------------\n// Begin Main Program\nconsole.log(\"JS Started, waiting for Python to start...\");\n// One-time initialization should be performed before syncing to Py\nshared.writeXF(\"nestedObj\", undefined);\n\n// Initial synchronization with Python\n// Write a value to empty memory and mark it full\nshared.writeEF(\"JS hello\", \"from Javascript\");\n\n// Wait for Python to start\nconsole.log(\"Hello \" + shared.readFE(\"Py hello\"));\n\n/*\n This synchronous exchange was a \"barrier\" between the two processes.\n\n The idiom of exchanging messages by writeEF and readFE constitutes\n a synchronization point between two processes. A barrier synchronizing\n N tasks would require N² variables, which is a reasonable way to\n implement a barrier when N is small. For larger numbers of processes\n barriers can be implemented using shared counters and the Fetch-And-Add\n (FAA) EMS instruction.\n\n The initialization of EMS values as \"empty\" occurs when the EMS array\n was created with:\n \"setFEtags\": \"empty\",\n \"doSetFEtags\": true,\n If it becomes necessary to reset a barrier, writeXE() will\n unconditionally and immediately write the value and mark it empty.\n */\n\n\n// Convenience function to synchronize with another process\n// The Python side reverses the barrier names\nfunction barrier(message) {\n console.log(\"Entering Barrier:\", message);\n shared.writeEF(\"py side barrier\", undefined);\n shared.readFE(\"js side barrier\");\n console.log(\"Completed Barrier.\");\n console.log(\"---------------------------------------------------\");\n}\n\n\n// --------------------------------------------------------------------\nbarrier(\"Trying out the barrier utility function\");\n// --------------------------------------------------------------------\n\n\n// JS and Python EMS can read sub-objects with the same syntax as native objects,\n// but writes are limited to only top level attributes. This corresponds to the\n// implied \"emsArray.read(key).subobj\" performed\nshared.top = {};\nconsole.log(\"Assignment to top-level attributes works normally:\", shared.top);\nshared.top.subobj = 1; // Does not take effect like top-level attributes\nconsole.log(\"But sub-object attributes do not take effect. No foo?\", shared.top.subobj);\n\n// A nested object can be written at the top level.\nshared.nestedObj = {\"one\":1, \"subobj\":{\"left\":\"l\", \"right\":\"r\"}};\nconsole.log(\"Nested object read references work normally\", shared.nestedObj.subobj.left);\n\n// A workaround to operating on nested objects\nlet nestedObjTemp = shared.nestedObj;\nnestedObjTemp.subobj.middle = \"m\";\nshared.nestedObj = nestedObjTemp;\n\n// The explicitly parallel-safe way to modify objects is similar to that \"workaround\"\nnestedObjTemp = shared.readFE(\"nestedObj\");\nnestedObjTemp.subobj.front = \"f\";\nshared.writeEF(\"nestedObj\", nestedObjTemp);\n\n\n// --------------------------------------------------------------------\nbarrier(\"This barrier matches where Python is waiting to use nestedObj\");\n// --------------------------------------------------------------------\n\n// Python does some work now\n// Initialize the counter for the next section\nshared.writeXF(\"counter\", 0);\n\n// --------------------------------------------------------------------\nbarrier(\"JS and Py are synchronized at the end of working with nestedObj\");\n// --------------------------------------------------------------------\n\n// ---------------------------------------------------------------------------\n// Use the EMS atomic operation intrinsics to coordinate access to data\n// This is 10x as many iterations as Python in the same amount of time\n// because V8's TurboFan compiler kicks in.\nfor (let count = 0; count < 1000000; count += 1) {\n let value = shared.faa(\"counter\", 1);\n if (count % 100000 == 0) {\n console.log(\"JS iteration\", count, \" Shared Counter=\", value)\n }\n}\n\n\n// --------------------------------------------------------------------\nbarrier(\"Waiting for Py to finish it's counter loop\");\n// --------------------------------------------------------------------\n\nconsole.log(\"The shared counter should be 11000000 ==\", shared.counter);\n\n\n\n// ---------------------------------------------------------------------------\n// Ready to earn your Pro Card?\n\n// Wait for Py to initialize the array+string\nbarrier(\"Wait until it's time to glimpse into the future of Javascript\");\nconsole.log(\"The array+string from Py:\", shared.arrayPlusString);\n\nbarrier(\"Waiting for shared.arrayPlusString to be initialized\");\nconsole.log(\"JS can do array+string, it produces a string:\", shared.arrayPlusString + \" world!\");\nshared.arrayPlusString += \" RMW world!\";\nconsole.log(\"JS performing \\\"array + string\\\" produces a self-consistent string:\", shared.arrayPlusString);\nbarrier(\"Arrive at the future.\");\n\n\n// ---------------------------------------------------------------------------\n// Leave a counter running on a timer\n// First re-initialize the counter to 0\nshared.writeXF(\"counter\", 0);\n\nfunction incrementCounter() {\n // Atomically increment the shared counter\n let oldValue = shared.faa(\"counter\", 1);\n\n // If the counter has been set to \"Stop\", clear the timer interval\n if (oldValue == \"stop\") {\n console.log(\"Timer counter was signaled to stop\");\n clearInterval(timerCounter);\n }\n\n // Show counter periodically\n if (oldValue % 100 == 0) {\n console.log(\"Counter =\", shared.counter);\n }\n}\n\nvar timerCounter = setInterval(incrementCounter, 10); // Increment the counter every 1ms \n\nbarrier(\"Welcome to The Future™!\");\n// Fall into Node.js event loop with Interval Timer running\n"
},
{
"path": "Examples/Interlanguage/interlanguage.py",
"content": "#!/usr/bin/env python\n# -*- coding: utf-8 -*-\n\"\"\"\n +-----------------------------------------------------------------------------+\n | Extended Memory Semantics (EMS) Version 1.4.5 |\n | Synthetic Semantics http://www.synsem.com/ mogill@synsem.com |\n +-----------------------------------------------------------------------------+\n | Copyright (c) 2017, Jace A Mogill. All rights reserved. |\n | |\n | Redistribution and use in source and binary forms, with or without |\n | modification, are permitted provided that the following conditions are met: |\n | * Redistributions of source code must retain the above copyright |\n | notice, this list of conditions and the following disclaimer. |\n | * Redistributions in binary form must reproduce the above copyright |\n | notice, this list of conditions and the following disclaimer in the |\n | documentation and/or other materials provided with the distribution. |\n | * Neither the name of the Synthetic Semantics nor the names of its |\n | contributors may be used to endorse or promote products derived |\n | from this software without specific prior written permission. |\n | |\n | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS |\n | \"AS IS\" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT |\n | LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR |\n | A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL SYNTHETIC |\n | SEMANTICS LLC BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, |\n | EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, |\n | PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR |\n | PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF |\n | LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING |\n | NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS |\n | SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. |\n | |\n +-----------------------------------------------------------------------------+\n\"\"\"\n\"\"\"\n tl;dr Start the Javascript first, then the Python\n --------------------------------------------------\n\n See interlanguage.js for more detailed description of starting processes\n sharing EMS memory\n\n NOTE ABOUT \"STUCK\" JOBS\":\n Python runtime can get \"stuck\" and may not respond to Control-C\n but can still be interrupted with Control-Z or Control-\\\n This \"stuck\" behavior is due to how the Python/C interface protects\n the Python runtime.\n\"\"\"\n\nimport time\nimport sys\nimport random\nsys.path.append(\"../../Python/\")\nimport ems\n\n# Initialize EMS: 1 process, no thread-core affinity, user provided parallelism\n# Initialize EMS with 1 process, no thread-CPU affinity,\n# \"user\" mode parallelism, and a unique namespace for EMS runtime\n# (\"pyExample\") to keep the JS program distinct from Javascript EMS\n# program also running.\nems.initialize(1, False, \"user\", \"pyExample\")\n\n\n# The EMS array attributes must be the same by all programs sharing the array,\n# with the exception of the the initial file creation which uses \"useExisting:False\"\n# instead of \"useExisting:True\"\nmaxNKeys = 100\nbytesPerEntry = 100 # Bytes of storage per key, used for key (dictionary word) itself\nshared = ems.new({\n 'useExisting': True, # Connect the EMS memory created by JS, do not create a new memory\n 'filename': \"/tmp/interlanguage.ems\", # Persistent EMS array's filename\n 'dimensions': maxNKeys, # Maximum # of different keys the array can store\n 'heapSize': maxNKeys * 100, # 100 bytes of storage per key, used for key (dictionary word) itself\n 'useMap': True # Use a key-index mapping, not integer indexes\n})\n\n\n# ------------------------------------------------------------------------------------------\n# Begin Main Program\nprint \"\"\"Start this program after the JS program.\nIf this Python appears hung and does not respond to ^C, also try ^\\ and ^Z\n----------------------------------------------------------------------------\n\"\"\"\n\n# Initial synchronization with JS\n# Write a value to empty memory and mark it full\nshared.writeEF(\"Py hello\", \"from Python\")\n\n# Wait for JS to start\nprint \"Hello \", shared.readFE(\"JS hello\")\n\n\ndef barrier(message):\n \"\"\"Define a convenience function to synchronize with another process\n The JS side reverses the barrier names\"\"\"\n global shared\n naptime = random.random() + 0.5\n time.sleep(naptime) # Delay 0.5-1.5sec to make synchronization apparent\n print \"Entering Barrier:\", message\n shared.writeEF(\"js side barrier\", None)\n shared.readFE(\"py side barrier\")\n print \"Completed Barrier after delaying for\", naptime, \"seconds\"\n print \"------------------------------------------------------------------\"\n\n\nprint \"Trying out the new barrier by first napping for 1 second...\"\ntime.sleep(1)\n# ---------------------------------------------------------------------\nbarrier(\"Trying out the barrier utility function\")\n# ---------------------------------------------------------------------\n\n\n# --------------------------------------------------------------------\n# This barrier matches where JS has set up \"shared.top\", and is now waiting\n# for Python to finish at the next barrier\nbarrier(\"Finished waiting for JS to finish initializing nestedObj\")\n\nprint \"Value of shared.nestedObj left by JS:\", shared.nestedObj\ntry:\n shared.nestedObj.number = 987 # Like JS, setting attributes of sub-objects will not work\n print \"ERROR -- This should not work\"\n exit()\nexcept:\n print \"Setting attributes of sub-objects via native syntax will not work\"\n\nprint \"These two expressions are the same:\", \\\n shared[\"nestedObj\"].read()[\"subobj\"][\"middle\"], \"or\", \\\n shared.read(\"nestedObj\")[\"subobj\"][\"middle\"]\n\n\n# --------------------------------------------------------------------\nbarrier(\"JS and Py are synchronized at the end of working with nestedObj\")\n# --------------------------------------------------------------------\n\n\n# Enter the shared counter loop\nfor count in xrange(100000):\n value = shared.faa(\"counter\", 1)\n if count % 10000 == 0:\n print \"Py iteration\", count, \" Shared counter=\", value\nbarrier(\"Waiting for Js to finish it's counter loop\")\n\n\n# --------------------------------------------------------------------\n# Ready to earn your Pro Card?\n\n# Make JS wait while Python demonstrates a \"create approach to language design\"\nshared.arrayPlusString = [1, 2, 3] # Initialize arrayPlusString before starting next phase\nbarrier(\"Wait until it's time to glimpse into the future of Python\")\ntry:\n dummy = shared.arrayPlusString + \"hello\"\n print \"ERROR -- Should result in: TypeError: can only concatenate list (not \\\"str\\\") to list\"\n exit()\nexcept:\n print \"Adding an array and string is an error in Python\"\n\nshared.arrayPlusString += \"hello\"\nprint \"However 'array += string' produces an array\", shared.arrayPlusString\n\n# Let JS know 'shared.arrayPlusString' was initialized\nbarrier(\"shared.arrayPlusString is now initialized\")\n# JS does it's work now, then wait for JS to do it's work\nbarrier(\"Arrive at the future.\")\n# JS does all the work in this section\nbarrier(\"Welcome to The Future™!\")\n\n\n# --------------------------------------------------------------------\n# JS is now in it's event loop with the counter running\nfor sampleN in xrange(10):\n time.sleep(random.random())\n print \"Shared counter is now\", shared.counter\n\n# Restart the counter \nshared.counter = 0\nfor sampleN in xrange(5):\n time.sleep(random.random() + 0.5)\n print \"Shared reset counter is now\", shared.counter\n\n\n# Wait 1 second and stop the JS timer counter\ntime.sleep(1)\nshared.counter = \"stop\"\n\n# Show the counter has stopped. Last value = \"stop\" + 1\nfor sampleN in xrange(5):\n time.sleep(random.random() + 0.5)\n print \"Shared reset counter should have stopped changing:\", shared.counter\n\nprint \"Exiting normally.\"\n"
},
{
"path": "Examples/KeyValueStore/README.md",
"content": "# Key-Value Store Example\n\nThis example implements a simple REST server, the API is a 1-to-1 \nmapping of REST commands to EMS commands, \neffectively making the EMS shared memory model\nan object store service over the network.\nThe RESTful API inherits all the atomicity\nand persistence properties of the underlying EMS array.\n\n## Parallelism & Deadlocks\n\nThe HTTP web interface used is Node's built-in HTTP module which\nimplements a serial event loop to process HTTP requests one at\ntime in a synchronous (blocking) fashion.\nThe execution time of EMS operations is similar to any other variable reference,\nmaking EMS most efficient as a synchronous programming model,\nhowever, because a `readFE` request may block the Node event loop\nclients susceptible to deadlock.\nA `writeEF` request that would unblock execution cannot handled \nby the single-threaded Node HTTP server that is blocked on a\n`readFE`, effectively deadlocking the system.\n\nThe two potential solutions, asynchronous execution and multi-processing, \nare complementary forms of parallelism that may be used\nto address the deadlock problem:\n\n- __Asynchronous Execution__:![](\"../Docs/streams.svg\")
\n![](\"../Docs/parWebServer.svg\")
\nEMS is a memory model and has no daemon process to monitor a network interface\nfor operations to execute.\nThis example builds on a standard HTTP server to implement\nshared memory parallelism within a single web request.\n\n## EMS as a Key Value Store\nClick here for the [EMS Key-Value Store Example](https://github.com/SyntheticSemantics/ems/tree/master/Examples/KeyValueStore)\n\nAn EMS array is used as a Key-Value store with a 1-to-1 mapping of REST operations \nto EMS memory operations.\nThe KV server is implemented using the Node.js built-in `Cluster` module to\ndemonstrate parallelism sources outside of EMS.\n\n\n## Work Queues\nThe parameters of the experiment are defined in the `initializeSharedData()` function.\nAn experiment creates several new EMS arrays and a work queue of \nrandomly generated transactions that update one or more of the arrays.\nParallel processes dequeue the work and perform atomic updates on \nthe new EMS arrays. Finally, the experiment performs checksums to ensure\nall the transactions were performed properly.\n\n```javascript\n arrLen = 1000000; // Maximum number of elements the EMS array can contain\n heapSize = 100000; // Amount of persistent memory to reserve for transactions\n nTransactions = 1000000; // Number of transactions to perform in the experiment\n nTables = 6; // Number of EMS arrays to perform transactions across\n maxNops = 5; // Maximum number of EMS arrays to update during a transaction\n```\n\n### `concurrent_Q_and_TM.js`\n```bash\nnode concurrent_Q_and_TM.js 4 # 1 process enqueues then processes work, 3 processes perform work\n````\nAll processes are started, one enqueues transactions while the others dequeue \nand perform the work. When all the work has been enqueued, that process also\nbegins dequeing and performing work.\n\n![](\"../Docs/tm_from_q.svg\")
\n\n### `workQ_and_TM.js`\n```bash\nnode workQ_and_TM.js 4 # Enqueue all the work, all 4 processes deqeue and perform work\n````\nAll the transactions are enqueued, then all the processes begin \nto dequeue and perform work until no work remains in the queue.\n\n\n\n## [Word Counting](#Word-Count)\nMap-Reduce is often demonstrated using word counting because each document can\nbe processed in parallel, and the results of each document's dictionary reduced\ninto a single dictionary.\nIn contrast with Map-Reduce implementations using a \ndictionary per document, the EMS implementation \nmaintains a single shared dictionary, eliminating the need for a reduce phase.\n\nThe word counts are sorted in parallel and the most frequently appearing words\nare printed with their counts.\n\n\n### Word Counting Using Atomic Operations\nA parallel word counting program is implemented in Python and Javascript.\nA shared dictionary is used by all the tasks, updates are made using\natomic operations. Specifically, `Compare And Swap (CAS)` is used to\ninitialize a new word from undefined to a count of 1. If this CAS\nfails, the word already exists in the dictionary and the operation\nis retried using `Fetch and Add`, atomically incrementing the count.\n\n### Forming the \"Bag of Words\" with Word Counts\n```javascript\nvar file_list = fs.readdirSync(doc_path);\nvar splitPattern = new RegExp(/[ \\n,\\.\\\\/_\\-\\<\\>:\\;\\!\\@\\#\\$\\%\\&\\*\\(\\)=\\[\\]|\\\"\\'\\{\\}\\?\\—]/);\n// Iterate over all the files in parallel\nems.parForEach(0, file_list.length, function (fileNum) {\n var text = fs.readFileSync(doc_path + file_list[fileNum], 'utf8', \"r\");\n var words = text.replace(/[\\n\\r]/g, ' ').toLowerCase().split(splitPattern);\n // Sequentially iterate over all the words in one document\n words.forEach(function (word) {\n wordCounts.faa(word, 1); // Atomically increment the count of times this word was seen\n });\n});\n```\n\n### Sorting the Word Count list by Frequency\nParallel sorting of the word count is implemented as a multi-step process:\n\n1. The bag of words is processed by all the processess, each process\n building an ordered list of the most common words it encounters\n2. The partial lists from all the processes are concatenated into a single list\n3. The list of the most common words seen is sorted and truncated\n\n```javascript\nvar biggest_counts = new Array(local_sort_len).fill({\"key\": 0, \"count\": 0});\nems.parForEach(0, maxNKeys, function (keyN) {\n var key = wordCounts.index2key(keyN);\n if (key) {\n // Perform an insertion sort of the new key into the biggest_counts\n // list, deleting the last (smallest) element to preserve length.\n var keyCount = wordCounts.read(key);\n var idx = local_sort_len - 1;\n while (idx >= 0 && biggest_counts[idx].count < keyCount) { idx -= 1; }\n var newBiggest = {\"key\": key, \"count\": keyCount};\n if (idx < 0) {\n biggest_counts = [newBiggest].concat(biggest_counts.slice(0, biggest_counts.length - 1));\n } else if (idx < local_sort_len) {\n var left = biggest_counts.slice(0, idx + 1);\n var right = biggest_counts.slice(idx + 1);\n biggest_counts = left.concat([newBiggest].concat(right)).slice(0, -1);\n }\n }\n});\n// Concatenate all the partial (one per process) lists into one list\nstats.writeXF('most_frequent', []); // Initialize the list\nems.barrier(); // Wait for all procs to have finished initialization\nstats.writeEF('most_frequent', stats.readFE('most_frequent').concat(biggest_counts));\nems.barrier(); // Wait for all procs to have finished merge\nems.master(function() { // Sort & print the list of words, only one process is needed\n biggest_counts = stats.readFF('most_frequent');\n biggest_counts.sort(function (a, b) { return b.count - a.count; });\n // Print only the first \"local_sort_len\" items -- assume the worst case\n // of all the largest counts are discovered by a single process\n console.log(\"Most frequently appearing terms:\");\n for (var index = 0; index < local_sort_len; index += 1) {\n console.log(index + ': ' + biggest_counts[index].key + \" \" + biggest_counts[index].count);\n }\n});\n```\n\n### Word Count Performance\nThis section reports the results of running the Word Count example on\ndocuments from Project Gutenberg.\n2,981,712,952 words in several languages were parsed, totaling 12,664,852,220 bytes of text.\n\n![](\"../Docs/wordcount.svg\")
\n\n\n\n## Inter-Language Programming\n[Inter-Language Shared Memory Programming](https://github.com/SyntheticSemantics/ems/tree/master/Examples/Interlanguage)\n\nThe programs `interlanguage.js` and `interlanguage.py` demonstrate sharing\nobjects between Javascript and Python.\nA variety of synchronization and execution models are shown.\n\n\n### Example\n![](\"../Docs/ems_js_py.gif\")
\n\n###### Copyright (C)2011-2019 Jace A Mogill\n"
},
{
"path": "Examples/STREAMS/README.md",
"content": "# EMS.js STREAMS Bandwidth Benchmark\n\nThe original [STREAMS](https://www.cs.virginia.edu/stream/)\nbenchmark measures the _Speed Not To Exceed_ \nbandwidth of a CPU to it's attached RAM by performing large\nnumbers of simple operations. This version for\nEMS![](\"../../Docs/ParallelContextsBSP.svg\")
\n![](\"../../Docs/ParallelContextsFJ.svg\")
\n![](\"../../Docs/streams.svg\")
\n![](\"../../Docs/parWebServer.svg\")
\n\n## REST API\n\nA valid request URL consists of path naming the session ID\nand a query that specifies if the session is new/restarting, or\nis a continuation of a previous session.\n \n__REST API:__ `/sessionID?![](\"../../Docs/tasksAndLoopsFJ.svg\")
\n ![](\"../../Docs/tasksAndLoopsBSP.svg\")
\n ![](\"Docs/ems_js_py.gif\")
\n\nInter-language example in [interlanguage.{js,py}](https://github.com/mogill/ems/tree/master/Examples/Interlanguage)\nThe animated GIF demonstrates the following steps:\n* Start Node.js REPL, create an EMS memory\n* Store \"Hello\"\n* Open a second session, begin the Python REPL\n* Connect Python to the EMS shared memory\n* Show the object created by JS is present in Python\n* Modify the object, and show the modification can be seen in JS\n* Exit both REPLs so no programs are running to \"own\" the EMS memory\n* Restart Python, show the memory is still present\n* Initialize a counter from Python\n* Demonstrate atomic Fetch and Add in JS\n* Start a loop in Python incrementing the counter\n* Simultaneously print and modify the value from JS\n* Try to read \"empty\" data from Python, the process blocks\n* Write the empty memory, marking it full, Python resumes execution\n\n## Types of Concurrency\n![](\"Docs/typesOfParallelism.svg\")
\n ![](\"Docs/ParallelContextsBSP.svg\")
\n ![](\"Docs/ParallelContextsFJ.svg\")
\n ![](\"Docs/wordcount.svg\")
\n\nThe performance of this program was measured using an Amazon EC2 instance:![](\"Docs/streams.svg\")
\n\n\n### Benchmarking of Transactions and Work Queues\n[Transactions and Work Queues Example](https://github.com/mogill/ems/tree/master/Examples)\n\nTransactional performance is measured alone, and again with a separate\nprocess appending new processes as work is removed from the queue.\nThe experiments were run using an Amazon EC2 instance:![](\"Docs/tm_no_q.svg\")
\n ![](\"Docs/tm_from_q.svg\")
\n ![](\"Docs/tm_no_q_str.svg\"/)
\n ![](\"Docs/memLayoutLogical.svg\")
\n ![](\"Docs/fsmSimple.svg\")
\n ![](\"Docs/blockDiagram.svg\")
\n