Extended Memory Semantics (EMS) complements serial programming
models with
transactional and other fine-grained synchronization capabilities
to support parallel programming.
Much of the challenge in implementing distributed and parallel
programs derives from finding, marshaling, and synchronizing
data. Extended Memory Semantics (EMS) unifies these tasks into a
single programming and execution model. EMS implements a shared
address space with a rich set of primitives for parallel access of data
structures. It is not a source of parallelism itself, instead it
complements other parallel programming models and integrates shared
memory data access and synchronization.
EMS leverages existing tool chains instead of replacing them and is
compatible with legacy applications, libraries, frameworks,
operating systems, and hardware.
Because EMS represents
memory and not processes, it may persist independently of any
application, and it's state may be replicated, archived, or forked.
Applications may attach and detach from the memory in much the
same way applications use a shared database or filesystem.
Synchronization As a Property of the Data, Not a Duty for Tasks
EMS internally stores tags that are used for synchronization of
user data, allowing synchronization to happen independently of
the number or kind of processes accessing the data. The tags
can be thought of as being in one of three states, Empty,
Full, or Read-Only, and the EMS primitives enforce
atomic access through automatic state transitions.
EMS Data Tag Transitions & Atomic operations:
F=Full, E=Empty, X=Don't Care, RW=Readers-Writer lock (# of current readers)
CAS=Compare-and-Swap, FAA=Fetch-and-Add
The function name readFE means "Read when full and mark empty",
writeEF means "Write when empty and mark full",
writeXF means "Write unconditionally and mark
full", etc. In the most simple case, full-empty tags are used
to block readers until memory is marked full by a writer thread
that itself blocks until the memory is marked empty. This is
effectively a dataflow or producer-consumer execution model that
may involve any number of producers and consumers.
EMS memory is an array of JSON primitive values
(Number, Boolean, String, or Undefined) accessed using atomic
operators and/or transactional memory. Safe parallel access
is managed by passing through multiple gates: First mapping a
key to an index, then accessing user data protected by EMS
tags, and completing the whole operation atomically.
The EMS array may be indexed directly using an integer, or using a key
mapping of any primitive type. When a map is used, the key and data
itself are updated atomically.
The full-empty primitives are
used construct other thread-safe data types such as
atomic read-modify-write operations and Transactional Memory (TM).
Principles of Operation
When the require('ems')(...) statement is executed by a program,
EMS first creates a shared memory region to rendezvous and
communicate with other EMS threads, then,
using the built-in
fork primitives,
creates the additional threads executing
using one of two execution models: fork-join or Bulk Synchronous Parallel (BSP).
BSP invokes the same script as the master thread (found in process.argv[2]),
whereas fork-join execution invokes parallel region around a function.
Under BSP, all threads execute the entire program unless statements are explicitly skipped.
Fork-join parallelism has a single entry point and executes sequentially
until a parallel region is started with ems.parallel( func ).
Fork-Join parallelism follows the traditional single-threaded execution
model until a parallel region where threads are dynamically added to
perform iterations of a loop. Under BSP parallelism every thread
enters the program at the main entry point.
Fork-join creates parallel regions
much like OpenMP's #pragma omp parallel directive.
Under BSP, all threads enter the main program and execute all
statements, synchronizing at barriers.
In addition to ordinary sequential loops,
within a parallel region ems.parForEach( func )
loops distribute iterations among the threads
using several load balancing scheduling options.
The master thread preserves all the characteristics and capabilities of
an ordinary
job, and all legacy applications, modules, packages, frameworks, and test apparatus
will work normally. Software that does not use EMS is not affected by it's presence.
Atomic operations like compare-and-swap (CAS) and fetch-and-add (FAA)
that are typically non-blocking will block if the full/empty
tag is set to empty.
Stack/queue operators are deadlock free,
blocking operations and should be thought of as thread-safe but not concurrent.
EMS transactions are also deadlock free and support element-level locking
for the highest possible currency.
Dataflow programs directly manipulating the full/empty tags
may deadlock if a program attempts to re-acquire a lock
already held, or acquire locks in a different order than other threads.
EMS programs may be run with any number of threads, including
single threaded and over-subscribed.
A logical overview of what program statements cause threads to be created
and how shared data is referenced.
Performance
These experiments were run on an Amazon EC2 instance: cr1.8xlarge: 244 GiB memory, 88 EC2 Compute Units, 240 GB of local instance storage, 64-bit platform, 10 Gigabit Ethernet
Using the general transactional memory capabilities to process
randomly generated operations stored in a shared work queue.
Transaction processing but generating the work from a loop instead of reading it from a shared queue.
Word Count of documents from Project Gutenberg in a variety of languages. Average document was
about 250kb in length.
Built-In Composed Operations and Parallel Data Structures
High-level data abstractions can be constructed from the EMS primitives,
and EMS itself
composes the primitives to implement transactional memory (TM), stacks, and queues.
User defined composed operations can be added to EMS classes just as new methods
are added to other JavaScript objects.
Transactional Memory
Transactional Memory (TM) provides atomic access to multiple shared objects
in a manner similar to transactional databases. EMS implements mutual exclusion
on specific data elements using the Full/Empty tags,
and shared read-only access with a multiple readers-single writer tag.
Stacks and Queues
Parallel-safe stacks and queues are built-in intrinsics based on Full/Empty tags.
Stacks and queues are by definition serial data structures and do
not support any concurrency. Although highly efficient, a shared
resource like these can become a hot-spot when dozens of threads compete
for access.
Types of Parallelism
EMS Data Tag Transitions - The four data element states
and the intrinsic EMS atomic operations to transition
between them.
Why Shared Memory Parallelism?
Multithreading complements other forms of parallelism and can be
combined with other forms of concurrency for multiplicative benefits.
Contrary Notions of Strong & Weak Scaling
Strong Scaling
Weak Scaling
Scaling Goal
Solve the same problem, only faster
Solve a bigger problem in the same amount of time
Problem Size
Stays constant while number of processors increases
Grows with the number of processors
Scaling is limited by
Inter-process communication
Problem size
Resiliency
Single failure causes entire job to fail,
SLAs achieved through efficient checkpoint-restart.
Failed sub-tasks are detected and retried. SLAs achieved through fault resiliency.
Programming Models
MPI, GASNet, Chapel, X10, Co-Array Fortran, UPC
Batch jobs, Map-Reduce
Historical Precedents for Data-Centric Multithreading
EMS builds on three experimental computer architectures from the
1980's: the NYU Ultracomputer, the MIT J-Machine, and the Tera Multi-Threaded Architecture (MTA).
Specifically, the Ultra introduced combining networks as a basic
architectural feature,
the J-Machine made moving a task to data as easy as moving data to the processor,
and the MTA used massive multithreading to mask latency and
had fine-grained synchronization associated with the data, not tasks.
Bugs
The old data argument for CAS requires a deep compare -- this is possible but
entirely compatible with JS semantics which compares type, not contents.
Similarly, FAA does not implement add on object type data, but extending the
semantics of JS would make this sensible
Roadmap
Languages and APIs
In addition to JavaScript, Python, and C/C++, EMS can be added to other languages.
Languages that can share the EMS API:
- Scala
- PHP
- Java
- Perl
- Your name here
Examples, Benchmarks, Tests
Other programs included with the distribution for demonstration, test, and benchmarking purposes.
- Matrix multiply
- Graph500
- Sobel Filter
- MongoDB Replica Server
RiverTrail Extensions
A few years ago a proposed set of language extensions called RiverTrail
was being supported by Intel and was implemented as a Firefox Plug-In.
The hooks used by the extension are no longer supported in Firefox, and
the future of RiverTrail is unclear. To the extent possible, EMS should
implement RiverTrail extensions.
EMS internally stores tags that are used for synchronization of user data,
allowing synchronization to happen independently of the number or kind of processes accessing
the data.
The EMS primitives enforce atomic access using automatic state transitions,
and higher level intrinsics like stacks, queues, and transactional
memory are built on the EMS primitives.
EMS memory is an array of JSON values
accessed using atomic operators and/or transactional memory.
Safe parallel access is managed by passing through multiple gates:
First mapping a key to an index, then
accessing user data protected by EMS tags, and completing the whole
operation atomically.
EMS Data Tag Transitions & Atomic operations:
F=Full, E=Empty, X=Don't Care, RW=Readers-Writer lock (# of current readers)
CAS=Compare-and-Swap, FAA=Fetch-and-Add
EMS Class & Array Methods
Operations which inspect or affect the global EMS state are performed
using class methods of
the EMS object returned by the require statement.
Operations that modify the data and tags are performed using methods
belonging to EMS array to be operated on.
Initialize the EMS module, starting
all the other threads. Thread identity and processor affinity
is assigned when the thread is created.
ARGUMENTS
nThreads
<Number>
Total number of threads the job should use.
threadAffinity
<Boolean>
(Optional, Default = false, Affects only Linux.)
Set the scheduling affinity of each thread to it's own
core, assigning over-subscribed threads in a round-robin
fashion.
parallelType
<String>
(Optional, Default=bsp)
One of bsp, fj, or user. Execution model:
bsp will use EMS' built-in Bulk Synchronous Parallel execution,
fj uses EMS' built-in Fork-join execution
user creates no parallelism
contextName
<String>
(Optional, Default=anonymous)
Unique name of parallel context being initialized, required to distinguish
between multiple EMS parallel programs running simultaneously on the
same system.
RETURNS
ems = {
nThreads : Number, // Number of threads executing
myID : Number // This thread's ID 0..nThreads-1
}
EXAMPLES
ems = require('ems')(process.argv[2])
Use the first command line argument as the number of nodes:
node foo.js 4 executes using 4 threads.
Attach to an existing or create a new EMS array.
Creation of new (do not use existing) EMS memory regions implies a barrier,
whereas using an existing EMS file will block until the file exists.
ARGUMENTS
nElements
<Number>
(Optional, Default is 1) Maximum number of elements in the EMS array or map.
<Array>
An array of dimensions of a multi-dimensional array.
A 100×30×50 cube is described by [100, 30, 50].
heapSize
<Number>
(Optional, Default is 0)
Maximum number of bytes reserved for strings,
arrays, maps, and object elements in this array.
The actual amount of memory allocated for use is
rounded up to the nearest power of 2
(until a better memory allocator is implemented).
Additional memory is allocated for bookkeeping.
Memory block size is defined in ems_alloc.h: EMS_MEM_BLOCKSZbut should be configurable at create time.
fileName
<String>
(Optional, Default is anonymous) Fully qualified file name
of the file to use as a persistent backing store for the EMS array,
tags, and bookkeeping information.
emsArrayDescriptor
<Object>
(Alternative to two scalar argument) A complete EMS Array
descriptor may be passed as the only argument instead of
scalar arguments.
emsArrayDescriptor = {
dimensions : 100, // Required: Max # elements in EMS array
// Integer for 1-D, or array of dims [x, y, z]
heapSize : 100000, // Optional, default=0: Space, in bytes, for
// strings, maps, and objects. Rounded up to nearest power of two
mlock : 0, // Optional, default=0%: % of EMS memory to lock into RAM
useMap : true, // Optional, default=false: Map keys to indexes
useExisting : true, // Optional, default=false:
// Preserve data if an file already exists
persist : true, // Optional, default=true:
// Preserve the file after threads exit
doDataFill : false, // Optional, default=false: Initialize memory
dataFill : undefined, // Optional, If this property is defined,
// the EMS memory is filled with this value
setFEtags : 'full', // Optional, If defined, set 'full' or 'empty'
filename : '/path/to/file' // Optional, default=anonymous:
// Path to the persistent file of this array
}
RETURNS
<EMS Array>
EXAMPLES
var foo = ems.new(nItems)
Create a new non-persistent shared memory EMS
array with no heap space. Scalar data (Number, Boolean, Undefined)
may be stored in this array, but not strings, arrays, or objects.
var foo = ems.new(nItems, size,
'/tmp/EMS_foo')
Create a file-backed EMS shared memory
space with the filename /tmp/EMS_foo. In addition to the
scalar storage,
space for strings totaling
size bytes is also reserved for strings, arrays, objects, and maps.
var x = ems.new(nItems, size)
Create a new non-persistent shared memory EMS
array that has space for strings totaling size bytes
Parallel Region (Fork-Join execution only)
CLASS METHOD
ems.parallel( [args, ...] func )
SYNOPSIS
When using fork-join execution, the master
thread enters a parallel region by executing func
once from each process. The master process first starts all the
other processes running the function asynchronously, then executes the function
itself synchronously. All processes join (perform a barrier)
after completion of the work function.
The results of the function are discarded. Global variables on each node
are persistent between parallel regions.
ARGUMENTS
args
<Any>
Zero or more arguments to be passed to the function.
func
<Function>
Function to be executed once on every thread. The
optional arguments are used when calling the function. Return value is ignored.
EXAMPLES
ems.parallel( doWork )
The function doWork is executed by every thread.
ems.parallel( foo, "Smith", 123, doWork )
The function call doWork(foo, "Smith", 123)
is performed once by each process.
Parallel Loops
CLASS METHOD
ems.parForEach( first, last, function [, scheduling [, minChunk] ] )
SYNOPSIS
Parallel loop execution, distributing
the iterations among all the threads. The function is invoked
with one argument, the current iteration number. Iterations are
divided among the threads according to the
scheduling method specified.
Parallel for loops must not be nested.
The system should check for this and fall back on serial execution.
A barrier is implied at
the end.
ARGUMENTS
first
<Number>
Index to start iterating
last
<Number>
Index to stop iterating (non-inclusive).
func
<Function>
Loop body, only input argument is current loop index:
function foo(idx) {...}
scheduling
<String>
guided [minChunk]: Decreasing amounts of work are assigned to each task until minChunk iterations per thread is reached. Load balancing occurs when new chunks are assigned to threads. static: Equal number of iterations are given to each thread, no dynamic load balancing is performed. dynamic: All threads share one index which is atomically
incremented by 1 after each iteration. Provides ideal load balancing at the
cost of high per-iteration overhead.
minChunk
<Number>
(Optional, only used when scheduling='guided', default=1) Minimum number of iterations assigned to a single thread.
EXAMPLES
ems.parForEach(0, nItems-1, func)
Execute the func function nItems-1 times with indexes
0..nItems-1, inclusive.
ems.parForEach(10000, 20000, func,
'guided', 200)
Distribute iterations numbered 10,000-20,000 (inclusive) using the guided
method with a minimum chunk size of 200 iterations
ems.parForEach(0, 999,
func, 'static')
Execute func() 1,000 times with indexes
0..999 inclusive. Distribute the iterations evenly across
threads in contiguous blocks.
Barrier Synchronization
CLASS METHOD
ems.barrier()
SYNOPSIS
All the threads must reach the same
barrier before proceeding. Failure to call a barrier from every
process will result in deadlock.
If called outside a parallel region, a barrier has no effect.
EXAMPLES
ems.barrier()
All threads reach the barrier before proceeding.
Critical Region
CLASS METHOD
ems.critical( func )
SYNOPSIS
Perform function func()
mutually exclusive of other threads. Serializes execution through
all critical regions.
Named regions would be more like OpenMP
ARGUMENTS
func
<Function>
Function to perform sequentially.
EXAMPLES
ems.critical( function() {
// Use a shared resources
} )
A shared resource is accessed sequentially, but in no particular order.
Execute on Master Thread Only
CLASS METHOD
ems.master( func )
SYNOPSIS
Perform function func()
only on thread 0, implies a barrier.
Perform function func()
only once by the first thread to reach the statement.
Implies a barrier.
ARGUMENTS
func
<Function>
Function to be performed once.
EXAMPLES
ems.single( function() {
console.log("Only first task")
} )
Console logging is performed only once.
Print Diagnostic Message
CLASS METHOD
ems.diag( message )
SYNOPSIS
Print a diagnostic message to the console with a prefix indicating the task ID.
ARGUMENTS
message
<String>
Text of message to print to the console
EXAMPLES
ems.diag( "Hello, world!" )
EMS 3: Hello, world! appears on console
EMS Array Methods
Intrinsic Atomic Operations (AMOs)
Read EMS Memory
ARRAY METHOD
emsArray.read( index )
emsArray.readFE( index )
emsArray.readFF( index )
emsArray.readRW( index ), emsArray.releaseRW( index )
SYNOPSIS
The read family of EMS
memory operations return the data stored in an EMS array element.
The value may be any JSON type.
read
Immediately and unconditionally returns the
stored value, ignoring the tag state.
Reading uninitialized mapped indexes
will return undefined regardless of the default value
that would be returned with read__, cas, faa.
readFE
Blocks until the data element is full, then
atomically reads the value and marks it empty.
readFF
Blocks until the data element is full, then
atomically reads leaving it full. This allows safe read access
of data which may be updated
simultaneously. readFF ensures mutual exclusion,
and will block if the data is already under a Readers-Writer
lock.
readRW, releaseRW
Blocks until the data element is full or
already under a Readers-Writer lock, then increments the
Readers-Writer reference count. The
function emsArray.releaseRW() decrements the reference
count, restoring the state to Full if no readers remain.
Read dictionary word at index/key idx in the dictionary array.
var x = arr.readFE(idx)
Block until the element at index i
is full, atomically read the value and mark it empty.
var x = arr.readFF(idx)
Block until the element at index i is full,
atomically read the value and leave it full.
var x = arr.readRW(idx)
Acquire a shared-exclusive readers-writer lock.
Write EMS Memory
ARRAY METHOD
emsArray.write( index, value )
emsArray.writeXE( index, value )
emsArray.writeXF( index, value )
emsArray.writeEF( index, value )
SYNOPSIS
Write a value to an element of an EMS array.
write
Immediately and unconditionally
writes the value to memory. This operation does not honor or
modify the full/empty tag status.
writeXE
Unconditionally and atomically writes the value to the data element and marks the element empty.
writeXF
Unconditionally and atomically writes the value to the data element and marks the element full.
writeEF
Blocks until the element is empty, and then atomically writes the value and marks the element full.
ARGUMENTS
index
<Number | String>
Index in the EMS array of data to read
value
<Any>
Primitive value to store in the array at element numbered index.
EXAMPLES
histogram.write(idx, 0)
Initialize the value of histogram[idx] to 0.
dictionary.write(idx, "Hello")
Add the string "Hello" to the EMS
array dictionary at index idx.
arr.writeXE(i, undefined)
Purge the memory at index i of the EMS array arr.
arr.writeXF(j, 'Mr. Jones')
Unconditionally write the
string 'Mr. Jones'to the EMS array arr
at index j and atomically mark the element full.
arr.writeEF(2, v)
Block until the element at index 2
of arr is empty, atomically write the
value v and mark the memory full.
Atomic Fetch and Add
ARRAY METHOD
emsArray.faa( index, value )
SYNOPSIS
Atomically read the array's JSON primitive element (scalar or string, not array or object),
add the value, and write the new value
back to memory. Return the original contents of the memory.
ARGUMENTS
index
<Integer | String>
Index of the element in the EMS array emsArray
to atomically add to.
value
< Number | Boolean | String | Undefined >
Value to add to the EMS memory.
RETURNS
< Number | Boolean | String | Undefined >
The results are the same type as if
a + b were performed.
EXAMPLES
oldVal =
statistics.faa( timerIdx, elapsed )
Return the value in memory before the add operation.
currentSum =
arr.faa( index, value ) + value
Return the current value after the atomic operation has occurred.
Atomic Compare and Swap
ARRAY METHOD
emsArray.cas( index, oldValue, newValue )
SYNOPSIS
Atomically read the JSON primitive element (scalar or string, not object or array)
stored at the array's index,
compare the original value to oldValue, and if they are equivalent
store the new value.
The CAS operation succeeded
if the value returned is equivalent
to oldValue.
CAS will block until the EMS memory is marked full. CAS is the
equivalent of atomically performing: if( arr[idx] == oldValue ) then arr[idx] = newValue
ARGUMENTS
index
<Integer | String>
Index into the EMS array to update.
oldValue
<Number | Boolean | String | Undefined>
Value to compare to the value stored in memory.
newValue
<Any >
Value to store if the value in memory is oldValue
RETURNS
< Number | Boolean | String | Undefined >
The value in memory when the compare was performed.
Evaluates as true if the lock stored
at arr[index] was acquired.
oldWord = users.cas(1234,
'Cooking', 'Eating')
Attempt to atomically update user 1234's record from the string
'Cooking' to the string 'Eating'
Composed Array Operations
Composed operations use EMS intrinsics to perform
deadlock free atomic operations
involving multiple EMS elements. The composed operations use the tags
and require data to be full or empty as appropriate for the semantics of the operation.
Transactional Processing of Multiple Elements
CLASS METHOD
ems.tmStart( tmElements )
ems.tmEnd( tmHandle, doCommit )
SYNOPSIS
Lock (by transitioning tags from Full to Empty)
one or more EMS elements in a deadlock
free way. When multiple locks must be acquired, this function
guarantees at least one thread will always make progress. The
optional third element indicates the element is read-only and will
not be modified by the task while the lock is held. Read-only
data is locked using a Readers-Writer lock, permitting additional concurrency.
Performing transactions within transactions can result in deadlock
if the thread tries to recursively lock an element.
ARGUMENTS
tmElements
<Array>
Array identifying which EMS array
elements should be locked. Each array element is itself an array
naming the EMS array and index/key of the data
and an optional Read-Only hint:
[ emsArray, index (, isReadOnly) ]
tmHandle
<Object>
Returned from tmStart(),
contains state information needed to abort or commit the transaction.
doCommit
<Boolean>
Commits the transaction if true,
or aborts and rolls back the transaction if false or undefined.
Acquire and free locks on the elements in lockList.
Element arrA[idxA0] may be modified, but elements
arrA[idxA1] and arrB[idxB0] are read-only.
ems.tmEnd( tm, true )
Commit the transaction
Stacks & Queues
ARRAY METHOD
emsArray.push( value )
emsArray.pop( )
emsArray.enqueue( value )
emsArray.dequeue( )
SYNOPSIS
Append or remove data from a LIFO or
FIFO. If the queue or stack is empty, the pop
or dequeue operation returns
Undefined, which is indistinguishable from an Undefined
that was explicitly pushed onto the stack.
ARGUMENTS
value
<Any>
Value to add to the queue or stack.
RETURNS
emsArray.pop(), emsArray.dequeue() : < Any >
emsArray.push() : < Number >
emsArray.enqueue() : < Number >
The number of elements presently on the stack or queue
EXAMPLES
comments.push( "Hello, world" )
Append the string to the EMS array comments
mostRecent = comments.pop()
Atomically return the value
at the top of the stack.
Look up a key used to map a value
CLASS METHOD
emsArray.index2key( index )
SYNOPSIS
Convert an index into an EMS array to the key used to map
a value to that hashed index. This function can be used
to iterate over all the elements of a mapped array.
ARGUMENTS
index
<Number>
Index of the element in the EMS array to get the key for
RETURNS
< Any >
The key used to map the value which hashed to this index.
Freeing EMS Arrays
CLASS METHOD
emsArray.destroy( remove_file )
SYNOPSIS
Release the persistent and non-persistent resources associated with an EMS array,
alternatively persisting the data if remove_file is false.
Implies a barrier.
ARGUMENTS
remove_file
<Boolean>
If true, remove the file, else allow the EMS file to persist.
RETURNS
None.
Synchronize EMS Memory to Persistent Storage
CLASS METHOD
emsArray.sync( [ index [, nElements] ] )
SYNOPSIS
Synchronize the EMS memory with persistent storage.
ARGUMENTS
index
<String | Number>
(Optional, default = entire EMS array)
Index of the element in the EMS array to synchronize to disk
nElements
<Number>
(Optional, only defined if index is also defined,
default = 1)
Number of sequential indexes, starting with index,
that should be synchronized to disk
RETURNS
< Boolean >
True if memory was successfully synchronized to disk,
otherwise false.
EXAMPLES
users.sync( userID )
The user's record is committed to disk before the function returns.
TODO Reduce
ARRAY METHOD
emsArray.reduce( func )
SYNOPSIS
Perform a parallel reduction on the elements of an EMS array.
ARGUMENTS
func
<Function>
Function to combine this element with the
partial reduction. Function arguments are func(element,
sum), and the function returns a new sum.
The arguments and results may be of any type.
RETURNS
< Any >
EXAMPLES
sum = distances.reduce(
function( val, sum ) {
return(sum+val) } )
Perform the arithmetic sum on the values in the array.
Reorder the array indexes to the
ordering specified in the
array order. If order contains the same
index more than once, the output for those indexes is
undefined.
ARGUMENTS
order
<Array>
Integer permutation array.
EXAMPLES
arr.permute( order )
Reorder the array arr
to the new element ordering.
TODO: Scan
ARRAY METHOD
emsArray.scan( func )
SYNOPSIS
Perform a parallel prefix operation on
the EMS array elements: a[i] = func(a[i], a[i-1])
ARGUMENTS
func
<Function>
Function to combine this element and the
previous element. Function arguments are func(thisElement, previousElement),
and returns a new partial result.
RETURNS
< Array >
EXAMPLES
partialSums = subtotals.scan(
function(val, sum ) { return(sum+val) } )
Return a EMS array with the partial sums in each element.
TODO: Map
ARRAY METHOD
emsArray.map( func [, name] )
SYNOPSIS
Produce a new EMS array of the same length
by invoking func()
for each element in the array to produce the corresponding element
in the new EMS array.
ARGUMENTS
func
<Function>
Function to perform on each array element. The function is invoked with one argument, the EMS primitive element, and returns a new primitive element.
Persistent filename
<String>
(Optional, default is non-persistent)
If the new mapping should have a persistent backing on a filesystem,
this assigns the fully qualified filename.
Scales each element in the array values from inches to cm, producing a new EMS array with scaled values.
TODO: Filter
ARRAY METHOD
emsArray.filter( func [, name] )
SYNOPSIS
Produce a new EMS array containing only the elements
which evaluate as true when passed as the argument to func(element).
The elements of the new array may appear in any order.
ARGUMENTS
func
<Function>
Function to perform on each array element. The function is invoked with one argument, the EMS primitive element, and returns a boolean indicating if the element should be included in the new array.
name
<String>
(Optional, default is non-persistent)
If the new mapping requires persistent backing on a filesystem,
the fully qualified filename is defined.
================================================
FILE: Examples/Interlanguage/README.md
================================================
# Inter-Language Persistent Shared Memory
In *The Future™*, memory will have new capabilities that
programmers will make decisions about how best to use:
- __Non-Volatility__ - No more "loading" files into variables in memory,
just use the same variables as last time.
- __Sharing__ - Directly sharing objects between programs instead of
communicating them through network messages.
- __Selectable Latency__ - Expensive fast memory for frequently accessed variables,
cheaper slower memory for less frequently used data.
- __Correctness__ - Very inexpensive bulk memory might give approximate values instead
of the exact value stored.
- __Location__ - Allocate memory in another another rack or data center
- __Mirroring__ - Automatic fail-over to backup memory
EMS presently supports the first two and is ready
to incorporate new memory capabilities as they become available.
## Sharing Objects Between Programs
Sharing EMS data between programs written in different languages
is no different than sharing data between programs in the same language,
or sharing data between same program run multiple times.
### Javascript Semantics in Other Languages
Every language has it's own semantics and storage sequences for it's data structures,
EMS makes the various representations interoperable using Javascript's semantics.
That is, if your Python program relies on horrifying semantics like:
```python
>>> a = [1,2,3]
>>> a + "abc"
Traceback (most recent call last):
File "", line 1, in
TypeError: can only concatenate list (not "str") to list
>>> a += "abc"
>>> a
[1, 2, 3, 'a', 'b', 'c']
>>>
```
You will be horrified to learn EMS will instead follow Javascript's horrifying semantics:
```javascript
> a = [1,2,3]
[ 1, 2, 3 ]
> a + 'abc'
'1,2,3abc'
> a += 'abc'
'1,2,3abc'
>
```
You may be nonplussed to learn this is *The Future™*.
I, too, was horrified and baffled at this discovery.
Alas, this is an inevitable truth of *The Future™*.
### Possible Orders for Starting Processes
Two processes sharing memory may start in any order, meaning programs
must take care when initializing to not overwrite another program which
may also be trying to initialize, or detect when it is necessary to
recover from a previous error.
| Persistent EMS Array File | (Re-)Initialize | Use |
| :------------- |:-------------:| :-----:|
| Already exists | A one-time initialization process truncates and creates the EMS array. The first program to start must create the EMS file with the `useExisting : false` attribute | Any number of processes may attach to the EMS array in any order using the attribute `useExisting : true` |
| Does not yet exist | Subsequent programs attach to the new EMS file with the `useExisting : true` attribute | N/A |
# Running the Example
The programs `interlanguage.py` and `interlanguage.js` are complementary
to each other, and must be started in the correct order:
__First start the Javascript program, then start the Python.__
```bash
dunlin> node interlanguage.js
JS Started, waiting for Python to start...
[ EXECUTION STALLS UNTIL PYTHON PROGRAM STARTS ]
Hello from Python
...
```
Second, start the Python program
```bash
dunlin> python interlanguage.py
Start this program after the JS program.
If this Python appears hung and does not respond to ^C, also try ^\ and ^Z
----------------------------------------------------------------------------
Hello from Javascript
...
```
Due to a well-intentioned signal mask in the Python runtime meant to
provide resiliency in the face of a failing native plug-in module,
sometimes Python will appear hung when it is blocked on an EMS operation.
In these situations an alternative way to kill the Python interpreter
is with SIGQUIT (Control-\ from most shells), or back-grounding the
job with Control-Z and sending the job a kill signal.
### What the Example is Doing
- Each process writes a hello message to an EMS variable, marking it full.
- Each process reads the variable written by the other process, blocking until it has been written
- A simple point-to-point barrier is demonstrated by handshaking
- Illustrate some of the differences between Python and Javascript nested objects
- Both JS and Py enter a loop that increments a shared counter
- JS checks the results to make sure every iteration occurred once
- Illustrate some quirks of language design in JS and Py
- JS leaves a shared counter running in the background via `setInterval`
- Python enters a loop that occasionally reads the shared counter, showing it being updated
- Python over-writes the counter with `stop`
- JS notices the counter has been replaced with `stop` and exits.
### Interactive Example
Illustration of sharing data between the Node.js and Python
done interactively from the REPLs.
================================================
FILE: Examples/Interlanguage/interlanguage.js
================================================
/*-----------------------------------------------------------------------------+
| Extended Memory Semantics (EMS) Version 1.4.5 |
| Synthetic Semantics http://www.synsem.com/ mogill@synsem.com |
+-----------------------------------------------------------------------------+
| Copyright (c) 2017, Jace A Mogill. All rights reserved. |
| |
| Redistribution and use in source and binary forms, with or without |
| modification, are permitted provided that the following conditions are met: |
| * Redistributions of source code must retain the above copyright |
| notice, this list of conditions and the following disclaimer. |
| * Redistributions in binary form must reproduce the above copyright |
| notice, this list of conditions and the following disclaimer in the |
| documentation and/or other materials provided with the distribution. |
| * Neither the name of the Synthetic Semantics nor the names of its |
| contributors may be used to endorse or promote products derived |
| from this software without specific prior written permission. |
| |
| THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS |
| "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT |
| LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR |
| A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL SYNTHETIC |
| SEMANTICS LLC BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, |
| EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, |
| PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR |
| PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF |
| LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING |
| NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS |
| SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. |
| |
+-----------------------------------------------------------------------------*/
/*
Start this JS program first, then start the Python program
Possible Orders for Starting Processes
---------------------------------------------
A persistent EMS file already exists
1. A one-time initialization process creates the EMS array
2. Any number of processes my attach to the EMS array in
any order.
A new EMS array will be created
1. The first program to start must create the EMS file with
the "useExisting : false" attribute
2. Subsequent programs attach to the new EMS file with the
"useExisting : true" attribute
*/
"use strict";
// Initialize EMS with 1 process, no thread-CPU affinity,
// "user" mode parallelism, and a unique namespace for EMS runtime
// ("jsExample") to keep the JS program distinct from Python EMS
// program also running.
let ems = require("ems")(1, false, "user", "jsExample");
if (Number(process.version[1]) < 6) {
console.log('Requires Node.js version 6.x or higher -- will not work with polyfill Harmony');
process.exit(1);
}
const maxNKeys = 100;
const bytesPerEntry = 100; // Bytes of storage per key, used for key (dictionary word) itself
let shared = ems.new({
"ES6proxies": true, // Enable native JS object-like syntax
"useExisting": false, // Create a new EMS memory area, do not use existing one if present
"filename": "/tmp/interlanguage.ems", // Persistent EMS array's filename
"dimensions": maxNKeys, // Maximum # of different keys the array can store
"heapSize": maxNKeys * bytesPerEntry,
"setFEtags": "empty", // Set default full/empty state of EMS memory to empty
"doSetFEtags": true,
"useMap": true // Use a key-index mapping, not integer indexes
});
//------------------------------------------------------------------------------------------
// Begin Main Program
console.log("JS Started, waiting for Python to start...");
// One-time initialization should be performed before syncing to Py
shared.writeXF("nestedObj", undefined);
// Initial synchronization with Python
// Write a value to empty memory and mark it full
shared.writeEF("JS hello", "from Javascript");
// Wait for Python to start
console.log("Hello " + shared.readFE("Py hello"));
/*
This synchronous exchange was a "barrier" between the two processes.
The idiom of exchanging messages by writeEF and readFE constitutes
a synchronization point between two processes. A barrier synchronizing
N tasks would require N² variables, which is a reasonable way to
implement a barrier when N is small. For larger numbers of processes
barriers can be implemented using shared counters and the Fetch-And-Add
(FAA) EMS instruction.
The initialization of EMS values as "empty" occurs when the EMS array
was created with:
"setFEtags": "empty",
"doSetFEtags": true,
If it becomes necessary to reset a barrier, writeXE() will
unconditionally and immediately write the value and mark it empty.
*/
// Convenience function to synchronize with another process
// The Python side reverses the barrier names
function barrier(message) {
console.log("Entering Barrier:", message);
shared.writeEF("py side barrier", undefined);
shared.readFE("js side barrier");
console.log("Completed Barrier.");
console.log("---------------------------------------------------");
}
// --------------------------------------------------------------------
barrier("Trying out the barrier utility function");
// --------------------------------------------------------------------
// JS and Python EMS can read sub-objects with the same syntax as native objects,
// but writes are limited to only top level attributes. This corresponds to the
// implied "emsArray.read(key).subobj" performed
shared.top = {};
console.log("Assignment to top-level attributes works normally:", shared.top);
shared.top.subobj = 1; // Does not take effect like top-level attributes
console.log("But sub-object attributes do not take effect. No foo?", shared.top.subobj);
// A nested object can be written at the top level.
shared.nestedObj = {"one":1, "subobj":{"left":"l", "right":"r"}};
console.log("Nested object read references work normally", shared.nestedObj.subobj.left);
// A workaround to operating on nested objects
let nestedObjTemp = shared.nestedObj;
nestedObjTemp.subobj.middle = "m";
shared.nestedObj = nestedObjTemp;
// The explicitly parallel-safe way to modify objects is similar to that "workaround"
nestedObjTemp = shared.readFE("nestedObj");
nestedObjTemp.subobj.front = "f";
shared.writeEF("nestedObj", nestedObjTemp);
// --------------------------------------------------------------------
barrier("This barrier matches where Python is waiting to use nestedObj");
// --------------------------------------------------------------------
// Python does some work now
// Initialize the counter for the next section
shared.writeXF("counter", 0);
// --------------------------------------------------------------------
barrier("JS and Py are synchronized at the end of working with nestedObj");
// --------------------------------------------------------------------
// ---------------------------------------------------------------------------
// Use the EMS atomic operation intrinsics to coordinate access to data
// This is 10x as many iterations as Python in the same amount of time
// because V8's TurboFan compiler kicks in.
for (let count = 0; count < 1000000; count += 1) {
let value = shared.faa("counter", 1);
if (count % 100000 == 0) {
console.log("JS iteration", count, " Shared Counter=", value)
}
}
// --------------------------------------------------------------------
barrier("Waiting for Py to finish it's counter loop");
// --------------------------------------------------------------------
console.log("The shared counter should be 11000000 ==", shared.counter);
// ---------------------------------------------------------------------------
// Ready to earn your Pro Card?
// Wait for Py to initialize the array+string
barrier("Wait until it's time to glimpse into the future of Javascript");
console.log("The array+string from Py:", shared.arrayPlusString);
barrier("Waiting for shared.arrayPlusString to be initialized");
console.log("JS can do array+string, it produces a string:", shared.arrayPlusString + " world!");
shared.arrayPlusString += " RMW world!";
console.log("JS performing \"array + string\" produces a self-consistent string:", shared.arrayPlusString);
barrier("Arrive at the future.");
// ---------------------------------------------------------------------------
// Leave a counter running on a timer
// First re-initialize the counter to 0
shared.writeXF("counter", 0);
function incrementCounter() {
// Atomically increment the shared counter
let oldValue = shared.faa("counter", 1);
// If the counter has been set to "Stop", clear the timer interval
if (oldValue == "stop") {
console.log("Timer counter was signaled to stop");
clearInterval(timerCounter);
}
// Show counter periodically
if (oldValue % 100 == 0) {
console.log("Counter =", shared.counter);
}
}
var timerCounter = setInterval(incrementCounter, 10); // Increment the counter every 1ms
barrier("Welcome to The Future™!");
// Fall into Node.js event loop with Interval Timer running
================================================
FILE: Examples/Interlanguage/interlanguage.py
================================================
#!/usr/bin/env python
# -*- coding: utf-8 -*-
"""
+-----------------------------------------------------------------------------+
| Extended Memory Semantics (EMS) Version 1.4.5 |
| Synthetic Semantics http://www.synsem.com/ mogill@synsem.com |
+-----------------------------------------------------------------------------+
| Copyright (c) 2017, Jace A Mogill. All rights reserved. |
| |
| Redistribution and use in source and binary forms, with or without |
| modification, are permitted provided that the following conditions are met: |
| * Redistributions of source code must retain the above copyright |
| notice, this list of conditions and the following disclaimer. |
| * Redistributions in binary form must reproduce the above copyright |
| notice, this list of conditions and the following disclaimer in the |
| documentation and/or other materials provided with the distribution. |
| * Neither the name of the Synthetic Semantics nor the names of its |
| contributors may be used to endorse or promote products derived |
| from this software without specific prior written permission. |
| |
| THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS |
| "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT |
| LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR |
| A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL SYNTHETIC |
| SEMANTICS LLC BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, |
| EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, |
| PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR |
| PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF |
| LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING |
| NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS |
| SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. |
| |
+-----------------------------------------------------------------------------+
"""
"""
tl;dr Start the Javascript first, then the Python
--------------------------------------------------
See interlanguage.js for more detailed description of starting processes
sharing EMS memory
NOTE ABOUT "STUCK" JOBS":
Python runtime can get "stuck" and may not respond to Control-C
but can still be interrupted with Control-Z or Control-\
This "stuck" behavior is due to how the Python/C interface protects
the Python runtime.
"""
import time
import sys
import random
sys.path.append("../../Python/")
import ems
# Initialize EMS: 1 process, no thread-core affinity, user provided parallelism
# Initialize EMS with 1 process, no thread-CPU affinity,
# "user" mode parallelism, and a unique namespace for EMS runtime
# ("pyExample") to keep the JS program distinct from Javascript EMS
# program also running.
ems.initialize(1, False, "user", "pyExample")
# The EMS array attributes must be the same by all programs sharing the array,
# with the exception of the the initial file creation which uses "useExisting:False"
# instead of "useExisting:True"
maxNKeys = 100
bytesPerEntry = 100 # Bytes of storage per key, used for key (dictionary word) itself
shared = ems.new({
'useExisting': True, # Connect the EMS memory created by JS, do not create a new memory
'filename': "/tmp/interlanguage.ems", # Persistent EMS array's filename
'dimensions': maxNKeys, # Maximum # of different keys the array can store
'heapSize': maxNKeys * 100, # 100 bytes of storage per key, used for key (dictionary word) itself
'useMap': True # Use a key-index mapping, not integer indexes
})
# ------------------------------------------------------------------------------------------
# Begin Main Program
print """Start this program after the JS program.
If this Python appears hung and does not respond to ^C, also try ^\ and ^Z
----------------------------------------------------------------------------
"""
# Initial synchronization with JS
# Write a value to empty memory and mark it full
shared.writeEF("Py hello", "from Python")
# Wait for JS to start
print "Hello ", shared.readFE("JS hello")
def barrier(message):
"""Define a convenience function to synchronize with another process
The JS side reverses the barrier names"""
global shared
naptime = random.random() + 0.5
time.sleep(naptime) # Delay 0.5-1.5sec to make synchronization apparent
print "Entering Barrier:", message
shared.writeEF("js side barrier", None)
shared.readFE("py side barrier")
print "Completed Barrier after delaying for", naptime, "seconds"
print "------------------------------------------------------------------"
print "Trying out the new barrier by first napping for 1 second..."
time.sleep(1)
# ---------------------------------------------------------------------
barrier("Trying out the barrier utility function")
# ---------------------------------------------------------------------
# --------------------------------------------------------------------
# This barrier matches where JS has set up "shared.top", and is now waiting
# for Python to finish at the next barrier
barrier("Finished waiting for JS to finish initializing nestedObj")
print "Value of shared.nestedObj left by JS:", shared.nestedObj
try:
shared.nestedObj.number = 987 # Like JS, setting attributes of sub-objects will not work
print "ERROR -- This should not work"
exit()
except:
print "Setting attributes of sub-objects via native syntax will not work"
print "These two expressions are the same:", \
shared["nestedObj"].read()["subobj"]["middle"], "or", \
shared.read("nestedObj")["subobj"]["middle"]
# --------------------------------------------------------------------
barrier("JS and Py are synchronized at the end of working with nestedObj")
# --------------------------------------------------------------------
# Enter the shared counter loop
for count in xrange(100000):
value = shared.faa("counter", 1)
if count % 10000 == 0:
print "Py iteration", count, " Shared counter=", value
barrier("Waiting for Js to finish it's counter loop")
# --------------------------------------------------------------------
# Ready to earn your Pro Card?
# Make JS wait while Python demonstrates a "create approach to language design"
shared.arrayPlusString = [1, 2, 3] # Initialize arrayPlusString before starting next phase
barrier("Wait until it's time to glimpse into the future of Python")
try:
dummy = shared.arrayPlusString + "hello"
print "ERROR -- Should result in: TypeError: can only concatenate list (not \"str\") to list"
exit()
except:
print "Adding an array and string is an error in Python"
shared.arrayPlusString += "hello"
print "However 'array += string' produces an array", shared.arrayPlusString
# Let JS know 'shared.arrayPlusString' was initialized
barrier("shared.arrayPlusString is now initialized")
# JS does it's work now, then wait for JS to do it's work
barrier("Arrive at the future.")
# JS does all the work in this section
barrier("Welcome to The Future™!")
# --------------------------------------------------------------------
# JS is now in it's event loop with the counter running
for sampleN in xrange(10):
time.sleep(random.random())
print "Shared counter is now", shared.counter
# Restart the counter
shared.counter = 0
for sampleN in xrange(5):
time.sleep(random.random() + 0.5)
print "Shared reset counter is now", shared.counter
# Wait 1 second and stop the JS timer counter
time.sleep(1)
shared.counter = "stop"
# Show the counter has stopped. Last value = "stop" + 1
for sampleN in xrange(5):
time.sleep(random.random() + 0.5)
print "Shared reset counter should have stopped changing:", shared.counter
print "Exiting normally."
================================================
FILE: Examples/KeyValueStore/README.md
================================================
# Key-Value Store Example
This example implements a simple REST server, the API is a 1-to-1
mapping of REST commands to EMS commands,
effectively making the EMS shared memory model
an object store service over the network.
The RESTful API inherits all the atomicity
and persistence properties of the underlying EMS array.
## Parallelism & Deadlocks
The HTTP web interface used is Node's built-in HTTP module which
implements a serial event loop to process HTTP requests one at
time in a synchronous (blocking) fashion.
The execution time of EMS operations is similar to any other variable reference,
making EMS most efficient as a synchronous programming model,
however, because a `readFE` request may block the Node event loop
clients susceptible to deadlock.
A `writeEF` request that would unblock execution cannot handled
by the single-threaded Node HTTP server that is blocked on a
`readFE`, effectively deadlocking the system.
The two potential solutions, asynchronous execution and multi-processing,
are complementary forms of parallelism that may be used
to address the deadlock problem:
- __Asynchronous Execution__:
Blocking calls are turned into continuations that will be scheduled
for execution again later.
The number of in-flight operations before potential deadlock is equal
to the number of continuations the runtime implements.
This is not presently implemented, and would require the addition
of new non-blocking variants of EMS functions, such as
`nbReadFE()` and `nbWriteEF()`.
- __Multi-Processing__:
Events are distributed across multiple single-threaded event loops.
Idle threads are assigned new events as they arrive, and each event
loop completes an event before beginning a new one. The number of
in-flight operations before potential deadlock is equal to the number
of processes.
This is implemented using Node's built-in [Cluster](https://nodejs.org/api/cluster.html) module.
## Starting the EMS Server process
Using Node's built-in `cluster` module that forks multiple Node.js
processes, we implement the multiple event loops needed for multi-processing.
Node's built-in `http` module provides the serial HTTP event loop which is
then parallelized using the `cluster` module.
`cluster` manages all child process creation, destruction, and communication.
Starting the web server from the command line:
```bash
node kv_store.js [# processes]
```
If the number of processes is not specified, 8 are started.
## Manually Sending Commands using CURL
An EMS array is presented as a Key-Value store with EMS memory operations
presented as a REST interface.
An ordinary Node.js Cluster is started to demonstrate parallelism outside of EMS.
```bash
node kv_store.js 8 # Start 8 servers, logs will appear on this terminal
```
```bash
dunlin> curl localhost:11080/writeXF?foo=Hello # Unconditionally write "Hello" to the key "foo" and mark Full
dunlin> curl localhost:11080/faa?foo=+World! # Fetch And Add returns original value: "Hello"
"Hello"
dunlin> curl localhost:11080/readFF?foo # Read Full and leave Full, final value is "Hello World!"
"Hello World!"
dunlin>
```
## Producer-Consumer
```bash
dunlin> curl localhost:11080/readFE?bar # EMS memory initialized as empty, this blocks
```
```bash
dunlin> curl localhost:11080/readFF?foo # Read Full and leave Full, final value is "Hello World!"
```
================================================
FILE: Examples/KeyValueStore/kv_store.js
================================================
/*-----------------------------------------------------------------------------+
| Extended Memory Semantics (EMS) Version 1.4.5 |
| Synthetic Semantics http://www.synsem.com/ mogill@synsem.com |
+-----------------------------------------------------------------------------+
| Copyright (c) 2011-2014, Synthetic Semantics LLC. All rights reserved. |
| Copyright (c) 2015-2017, Jace A Mogill. All rights reserved. |
| |
| Redistribution and use in source and binary forms, with or without |
| modification, are permitted provided that the following conditions are met: |
| * Redistributions of source code must retain the above copyright |
| notice, this list of conditions and the following disclaimer. |
| * Redistributions in binary form must reproduce the above copyright |
| notice, this list of conditions and the following disclaimer in the |
| documentation and/or other materials provided with the distribution. |
| * Neither the name of the Synthetic Semantics nor the names of its |
| contributors may be used to endorse or promote products derived |
| from this software without specific prior written permission. |
| |
| THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS |
| "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT |
| LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR |
| A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL SYNTHETIC |
| SEMANTICS LLC BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, |
| EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, |
| PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR |
| PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF |
| LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING |
| NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS |
| SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. |
| |
+-----------------------------------------------------------------------------*/
'use strict';
var ems = require('ems')(1, false, 'user');
var http = require('http');
var url_module = require("url");
var cluster = require('cluster');
var port = 8080;
var shared_counters;
var persistent_data;
var myID;
var persistent_data_options = {
dimensions: [100000],
heapSize: [10000000],
useExisting: false,
useMap: true,
filename: '/tmp/persistent_shared_web_data.ems'
};
var shared_counters_options = {
dimensions: [100],
heapSize: [10000],
useExisting: false,
useMap: true,
filename: '/tmp/counters.ems'
};
function handleRequest(request, response) {
var requestN = shared_counters.faa('nRequests', 1); // Increment the count of requests
var parsed_request = url_module.parse(request.url, true);
var key = Object.keys(parsed_request.query)[0];
var value = parsed_request.query[key];
if (value === '') { value = undefined; }
var data;
var opname;
if (!key) {
response.end("ERROR: No key specified in request");
return;
}
if (parsed_request.pathname.indexOf("/readFE") >= 0) {
data = persistent_data.readFE(key);
opname = 'readFE';
} else if (parsed_request.pathname.indexOf("/readFF") >= 0) {
data = persistent_data.readFF(key);
opname = 'readFF';
} else if (parsed_request.pathname.indexOf("/read") >= 0) {
data = persistent_data.read(key);
opname = 'read';
} else if (parsed_request.pathname.indexOf("/writeXE") >= 0) {
data = persistent_data.writeXE(key, value);
opname = 'writeXE';
} else if (parsed_request.pathname.indexOf("/writeXF") >= 0) {
data = persistent_data.writeXF(key, value);
opname = 'writeXF';
} else if (parsed_request.pathname.indexOf("/writeEF") >= 0) {
data = persistent_data.writeEF(key, value);
opname = 'writeEF';
} else if (parsed_request.pathname.indexOf("/write") >= 0) {
data = persistent_data.write(key, value);
opname = 'write';
} else if (parsed_request.pathname.indexOf("/faa") >= 0) {
data = persistent_data.faa(key, value);
opname = 'faa';
} else if (parsed_request.pathname.indexOf("/cas") >= 0) {
opname = 'cas';
var old_new_vals = value.split(',');
if (old_new_vals[0] === '') { old_new_vals[0] = undefined; }
if (old_new_vals[1] === '') { old_new_vals[1] = undefined; }
data = persistent_data.cas(key, old_new_vals[0], old_new_vals[1]);
} else {
data = "ERROR: No EMS command specified.";
}
var datastr = JSON.stringify(data);
console.log("Request #" + requestN + ", slave process " + myID + ": Op(" +
opname + ") key(" + key + ") val(" + value + ") data=" + datastr);
response.end(datastr);
}
if (cluster.isMaster) {
/* The Master Process creates the EMS arrays that the slave processes
* will attach to before creating the slave processes. This prevents
* parallel hazards if all slaves tried creating the EMS arrays.
*/
persistent_data_options.setFEtags = 'empty';
persistent_data_options.doSetFEtags = true;
persistent_data = ems.new(persistent_data_options);
shared_counters_options.dataFill = 0;
shared_counters_options.doDataFill = true;
shared_counters_options.setFEtags = 'full';
shared_counters_options.doSetFEtags = true;
shared_counters = ems.new(shared_counters_options);
// Seed the GUID generator with a unique starting value
shared_counters.writeXF('GUID', Math.floor(Math.random() * 10000000));
// All the one-time initialization is complete, now start slave processes.
// The number of processes is the limit of the number of pending requests
// before deadlock occurs.
var nProcesses = Number(process.argv[2]) || 8; // Number of "cluster" processes to start
for (var procnum = 0; procnum < nProcesses; procnum++) {
cluster.fork();
}
console.log(
"REST API: ?key[=value]\n" +
" EMSCommand: read, readFE, readFF, writeXE, writeXF, writeEF, faa, cas\n" +
" key: The index into the EMS array, must be a valid JSON element\n" +
" value: For write commands, the value stored in EMS.\n" +
" For CAS, the value is the old and new values separated by a comma:\n" +
" /cas?key=oldval,newval\n");
cluster.on('exit', function (worker, code, signal) {
console.log("worker " + worker.process.pid + "died");
});
} else {
/* Each Slave Cluster processes must connect to the EMS array created
* by the master process.
*/
persistent_data_options.useExisting = true;
persistent_data = ems.new(persistent_data_options);
shared_counters_options.useExisting = true;
shared_counters = ems.new(shared_counters_options);
myID = shared_counters.faa('myID', 1);
http.createServer(handleRequest).listen(port, function () {
console.log("Server " + myID + " listening on: http://localhost:" + port);
});
}
================================================
FILE: Examples/README.md
================================================
# Extended Memory Semantics (EMS) Examples
#### Table of Contents
* [Simple Loop Benchmarks](#Simple-Loop-Benchmarks)
* [Key-Value Store](#kv_store.js)
* [Work Queues](#Work-Queues)
* [Parallel Web Server](#web_server.js)
* [Word Counting](#Word-Count)
* [Implied EMS Operations](#harmony_proxies.js)
* [Inter-Language Programming](#Inter-language-Programming)
## Simple Loop Benchmarks
Click here for a __[Detailed Description of the EMS.js STREAMS implementation](https://github.com/SyntheticSemantics/ems/tree/master/Examples/STREAMS)__.
The original [STREAMS](https://www.cs.virginia.edu/stream/)
benchmark attempts to measure the _Speed Not To Exceed_
bandwidth of a CPU to it's attached RAM by performing large
numbers of simple operations on an array of integers or floats.
This version of STREAMS for EMS.js measures the rate of different atomic operations on
shared integer data, it is implemented twice using two different parallelization schemes:
Bulk Synchronous Parallel (BSP), and Fork-Join (FJ).
Experimental results are from the BSP version
on an AWS `c4.8xlarge (132 ECUs, 36 vCPUs, 2.9 GHz, Intel Xeon E5-2666v3, 60 GiB memory`.
## Parallelism Within a Single HTTP Request
Click here for the [Web Server Parallelism Example](https://github.com/SyntheticSemantics/ems/tree/master/Examples/WebServer)
EMS is a memory model and has no daemon process to monitor a network interface
for operations to execute.
This example builds on a standard HTTP server to implement
shared memory parallelism within a single web request.
## EMS as a Key Value Store
Click here for the [EMS Key-Value Store Example](https://github.com/SyntheticSemantics/ems/tree/master/Examples/KeyValueStore)
An EMS array is used as a Key-Value store with a 1-to-1 mapping of REST operations
to EMS memory operations.
The KV server is implemented using the Node.js built-in `Cluster` module to
demonstrate parallelism sources outside of EMS.
## Work Queues
The parameters of the experiment are defined in the `initializeSharedData()` function.
An experiment creates several new EMS arrays and a work queue of
randomly generated transactions that update one or more of the arrays.
Parallel processes dequeue the work and perform atomic updates on
the new EMS arrays. Finally, the experiment performs checksums to ensure
all the transactions were performed properly.
```javascript
arrLen = 1000000; // Maximum number of elements the EMS array can contain
heapSize = 100000; // Amount of persistent memory to reserve for transactions
nTransactions = 1000000; // Number of transactions to perform in the experiment
nTables = 6; // Number of EMS arrays to perform transactions across
maxNops = 5; // Maximum number of EMS arrays to update during a transaction
```
### `concurrent_Q_and_TM.js`
```bash
node concurrent_Q_and_TM.js 4 # 1 process enqueues then processes work, 3 processes perform work
````
All processes are started, one enqueues transactions while the others dequeue
and perform the work. When all the work has been enqueued, that process also
begins dequeing and performing work.
### `workQ_and_TM.js`
```bash
node workQ_and_TM.js 4 # Enqueue all the work, all 4 processes deqeue and perform work
````
All the transactions are enqueued, then all the processes begin
to dequeue and perform work until no work remains in the queue.
## [Word Counting](#Word-Count)
Map-Reduce is often demonstrated using word counting because each document can
be processed in parallel, and the results of each document's dictionary reduced
into a single dictionary.
In contrast with Map-Reduce implementations using a
dictionary per document, the EMS implementation
maintains a single shared dictionary, eliminating the need for a reduce phase.
The word counts are sorted in parallel and the most frequently appearing words
are printed with their counts.
### Word Counting Using Atomic Operations
A parallel word counting program is implemented in Python and Javascript.
A shared dictionary is used by all the tasks, updates are made using
atomic operations. Specifically, `Compare And Swap (CAS)` is used to
initialize a new word from undefined to a count of 1. If this CAS
fails, the word already exists in the dictionary and the operation
is retried using `Fetch and Add`, atomically incrementing the count.
### Forming the "Bag of Words" with Word Counts
```javascript
var file_list = fs.readdirSync(doc_path);
var splitPattern = new RegExp(/[ \n,\.\\/_\-\<\>:\;\!\@\#\$\%\&\*\(\)=\[\]|\"\'\{\}\?\—]/);
// Iterate over all the files in parallel
ems.parForEach(0, file_list.length, function (fileNum) {
var text = fs.readFileSync(doc_path + file_list[fileNum], 'utf8', "r");
var words = text.replace(/[\n\r]/g, ' ').toLowerCase().split(splitPattern);
// Sequentially iterate over all the words in one document
words.forEach(function (word) {
wordCounts.faa(word, 1); // Atomically increment the count of times this word was seen
});
});
```
### Sorting the Word Count list by Frequency
Parallel sorting of the word count is implemented as a multi-step process:
1. The bag of words is processed by all the processess, each process
building an ordered list of the most common words it encounters
2. The partial lists from all the processes are concatenated into a single list
3. The list of the most common words seen is sorted and truncated
```javascript
var biggest_counts = new Array(local_sort_len).fill({"key": 0, "count": 0});
ems.parForEach(0, maxNKeys, function (keyN) {
var key = wordCounts.index2key(keyN);
if (key) {
// Perform an insertion sort of the new key into the biggest_counts
// list, deleting the last (smallest) element to preserve length.
var keyCount = wordCounts.read(key);
var idx = local_sort_len - 1;
while (idx >= 0 && biggest_counts[idx].count < keyCount) { idx -= 1; }
var newBiggest = {"key": key, "count": keyCount};
if (idx < 0) {
biggest_counts = [newBiggest].concat(biggest_counts.slice(0, biggest_counts.length - 1));
} else if (idx < local_sort_len) {
var left = biggest_counts.slice(0, idx + 1);
var right = biggest_counts.slice(idx + 1);
biggest_counts = left.concat([newBiggest].concat(right)).slice(0, -1);
}
}
});
// Concatenate all the partial (one per process) lists into one list
stats.writeXF('most_frequent', []); // Initialize the list
ems.barrier(); // Wait for all procs to have finished initialization
stats.writeEF('most_frequent', stats.readFE('most_frequent').concat(biggest_counts));
ems.barrier(); // Wait for all procs to have finished merge
ems.master(function() { // Sort & print the list of words, only one process is needed
biggest_counts = stats.readFF('most_frequent');
biggest_counts.sort(function (a, b) { return b.count - a.count; });
// Print only the first "local_sort_len" items -- assume the worst case
// of all the largest counts are discovered by a single process
console.log("Most frequently appearing terms:");
for (var index = 0; index < local_sort_len; index += 1) {
console.log(index + ': ' + biggest_counts[index].key + " " + biggest_counts[index].count);
}
});
```
### Word Count Performance
This section reports the results of running the Word Count example on
documents from Project Gutenberg.
2,981,712,952 words in several languages were parsed, totaling 12,664,852,220 bytes of text.
## Inter-Language Programming
[Inter-Language Shared Memory Programming](https://github.com/SyntheticSemantics/ems/tree/master/Examples/Interlanguage)
The programs `interlanguage.js` and `interlanguage.py` demonstrate sharing
objects between Javascript and Python.
A variety of synchronization and execution models are shown.
### Example
###### Copyright (C)2011-2019 Jace A Mogill
================================================
FILE: Examples/STREAMS/README.md
================================================
# EMS.js STREAMS Bandwidth Benchmark
The original [STREAMS](https://www.cs.virginia.edu/stream/)
benchmark measures the _Speed Not To Exceed_
bandwidth of a CPU to it's attached RAM by performing large
numbers of simple operations. This version for
EMS.js measures the rate of different atomic operations on
shared integer data and is implemented using two different parallelization schemes:
Bulk Synchronous Parallel (BSP), and Fork-Join.
## Parallel Execution Models
The EMS.js STREAMS benchmark is implemented twice, once using Bulk Synchronous Parallelism
and again using Fork-Join parallelism.
BSP goes parallel at program startup,
decomposes loop iterations across processes, and performs a barrier at the end
of each loop. FJ starts serial and goes parallel for each loop, joining back
to serial execution before the next loop.
## Running the Benchmark
To run either version of STREAMS benchmark, specify the number of processes
as an argument to the program:
```bash
node streams_bulk_sync_parallel.js 8
```
Running the BSP version of the STREAMS benchmark reports the rate at
which each loop is performed, each loop exercising a single
EMS read and write primitive.
A second set of "Native" loops performs similar operations on ordinary
volatile private memory, taking advantage of all possible native
optimizations and just-in-time compilation.
These results show a performance difference of 20-50x
accessing EMS memory versus native memory,
which is still very favorable compared to the 1000-5000x difference accessing
a remote key-value store, or worse accessing magnetic media.
```bash
dunlin> time node streams_bulk_sync_parallel.js 3
EMStask 0: 1,000,000 write 6,896,551 ops/sec
EMStask 0: 1,000,000 writeXE 7,246,376 ops/sec
EMStask 0: 1,000,000 writeXF 7,246,376 ops/sec
EMStask 0: 1,000,000 read 7,042,253 ops/sec
EMStask 0: 1,000,000 reread 6,896,551 ops/sec
EMStask 0: 1,000,000 readFF 7,299,270 ops/sec
EMStask 0: 1,000,000 readFE 7,194,244 ops/sec
EMStask 0: 1,000,000 writeEF 7,299,270 ops/sec
EMStask 0: 1,000,000 copy 3,521,126 ops/sec
EMStask 0: 1,000,000 recopy 3,484,320 ops/sec
EMStask 0: 1,000,000 c=a*b 2,364,066 ops/sec
EMStask 0: 1,000,000 c+=a*b 1,715,265 ops/sec
EMStask 0: 1,000,000 checksum Infinity ops/sec
------------------------ START NATIVE ARRAY
EMStask 0: 1,000,000 write 333,333,333 ops/sec
EMStask 0: 1,000,000 rd/write 250,000,000 ops/sec
EMStask 0: 1,000,000 read 125,000,000 ops/sec
EMStask 0: 1,000,000 reread 250,000,000 ops/sec
EMStask 0: 1,000,000 copy 200,000,000 ops/sec
EMStask 0: 1,000,000 rmwcopy 250,000,000 ops/sec
EMStask 0: 1,000,000 c=a*b 100,000,000 ops/sec
EMStask 0: 1,000,000 c+=a*b 142,857,142 ops/sec
EMStask 0: 1,000,000 sum Infinity ops/sec
real 0m2.938s
user 0m8.190s
sys 0m0.172s
```
__NOTE:__ Adding timing instrumentation to the Fork-Join version is
left as an exercise to the reader.
## Benchmark Results
Benchmark results for the BSP implementation on an AWS
`c4.8xlarge (132 ECUs, 36 vCPUs, 2.9 GHz, Intel Xeon E5-2666v3, 60 GiB memory`.
================================================
FILE: Examples/STREAMS/streams_bulk_sync_parallel.js
================================================
/*-----------------------------------------------------------------------------+
| Extended Memory Semantics (EMS) Version 1.0.0 |
| Synthetic Semantics http://www.synsem.com/ mogill@synsem.com |
+-----------------------------------------------------------------------------+
| Copyright (c) 2011-2014, Synthetic Semantics LLC. All rights reserved. |
| Copyright (c) 2015-2016, Jace A Mogill. All rights reserved. |
| |
| Redistribution and use in source and binary forms, with or without |
| modification, are permitted provided that the following conditions are met: |
| * Redistributions of source code must retain the above copyright |
| notice, this list of conditions and the following disclaimer. |
| * Redistributions in binary form must reproduce the above copyright |
| notice, this list of conditions and the following disclaimer in the |
| documentation and/or other materials provided with the distribution. |
| * Neither the name of the Synthetic Semantics nor the names of its |
| contributors may be used to endorse or promote products derived |
| from this software without specific prior written permission. |
| |
| THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS |
| "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT |
| LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR |
| A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL SYNTHETIC |
| SEMANTICS LLC BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, |
| EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, |
| PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR |
| PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF |
| LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING |
| NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS |
| SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. |
| |
+-----------------------------------------------------------------------------*/
//
// Based on John D. McCalpin's (Dr. Bandwdith) STREAMS benchmark:
// https://www.cs.virginia.edu/stream/
//
//
// Also see the Fork-Join version in the Examples directory
// ===============================================================================
var ems = require('ems')(parseInt(process.argv[2]), false);
var arrLen = 1000000;
var a = ems.new(arrLen);
var b = ems.new(arrLen);
var c = ems.new(arrLen);
var x = new Array(arrLen);
var y = new Array(arrLen);
var z = new Array(arrLen);
//-------------------------------------------------------------------
// Timer function
function stopTimer(timer, nOps, label) {
function fmtNumber(n) {
var s = ' ' + n.toString().replace(/\B(?=(\d{3})+(?!\d))/g, ",");
if(n < 1) return n;
else { return s.substr(s.length - 15, s.length); }
}
ems.master( function() {
var now = new Date().getTime();
var x = (nOps*1000000) / ((now - timer) *1000);
ems.diag(fmtNumber(nOps) + label + fmtNumber(Math.floor(x).toString()) + " ops/sec");
} );
}
var timeStart = new Date().getTime();
ems.parForEach(0, arrLen, function(idx) { a.write(idx, idx) } );
stopTimer(timeStart, arrLen, " write ");
timeStart = new Date().getTime();
ems.parForEach(0, arrLen, function(idx) { a.writeXE(idx, idx) } );
stopTimer(timeStart, arrLen, " writeXE ");
timeStart = new Date().getTime();
ems.parForEach(0, arrLen, function(idx) { a.writeXF(idx, idx) } );
stopTimer(timeStart, arrLen, " writeXF ");
timeStart = new Date().getTime();
ems.parForEach(0, arrLen, function(idx) { a.read(idx, idx) } );
stopTimer(timeStart, arrLen, " read ");
timeStart = new Date().getTime();
ems.parForEach(0, arrLen, function(idx) { a.read(idx, idx) } );
stopTimer(timeStart, arrLen, " reread ");
timeStart = new Date().getTime();
ems.parForEach(0, arrLen, function(idx) { a.readFF(idx, idx) } );
stopTimer(timeStart, arrLen, " readFF ");
timeStart = new Date().getTime();
ems.parForEach(0, arrLen, function(idx) { a.readFE(idx, idx) } );
stopTimer(timeStart, arrLen, " readFE ");
timeStart = new Date().getTime();
ems.parForEach(0, arrLen, function(idx) { a.writeEF(idx, idx) } );
stopTimer(timeStart, arrLen, " writeEF ");
timeStart = new Date().getTime();
ems.parForEach(0, arrLen, function(idx) { b.writeXF(idx, a.readFF(idx)) } );
stopTimer(timeStart, arrLen, " copy ");
timeStart = new Date().getTime();
ems.parForEach(0, arrLen, function(idx) { b.writeXF(idx, a.readFF(idx)) } );
stopTimer(timeStart, arrLen, " recopy ");
timeStart = new Date().getTime();
ems.parForEach(0, arrLen, function(idx) { c.writeXF(idx, a.readFF(idx) * b.readFF(idx)) } );
stopTimer(timeStart, arrLen, " c=a*b ");
timeStart = new Date().getTime();
ems.parForEach(0, arrLen, function(idx) { c.writeXF(idx, c.readFF(idx) + (a.readFF(idx) * b.readFF(idx))) } );
stopTimer(timeStart, arrLen, " c+=a*b ");
timeStart = new Date().getTime();
stopTimer(timeStart, arrLen, " checksum ");
//===========================================================================================
var idx;
if(ems.myID == 0) {
console.log('------------------------ START NATIVE ARRAY');
timeStart = new Date().getTime();
for(idx = 0; idx < arrLen; idx++ ) { x[idx] = idx }
stopTimer(timeStart, arrLen, " write ");
timeStart = new Date().getTime();
for(idx = 0; idx < arrLen; idx++ ) { x[idx] += idx }
stopTimer(timeStart, arrLen, " rd/write");
timeStart = new Date().getTime();
var dummy = 0;
for(idx = 0; idx < arrLen; idx++ ) { dummy += x[idx] }
stopTimer(timeStart, arrLen, " read ");
timeStart = new Date().getTime();
for(idx = 0; idx < arrLen; idx++ ) { dummy += x[idx] }
stopTimer(timeStart, arrLen, " reread ");
timeStart = new Date().getTime();
for(idx = 0; idx < arrLen; idx++ ) { y[idx] = x[idx] }
stopTimer(timeStart, arrLen, " copy ");
timeStart = new Date().getTime();
for(idx = 0; idx < arrLen; idx++ ) { y[idx] += x[idx] }
stopTimer(timeStart, arrLen, " rmwcopy ");
timeStart = new Date().getTime();
for(idx = 0; idx < arrLen; idx++ ) { z[idx] = x[idx] * y[idx] }
stopTimer(timeStart, arrLen, " c=a*b ");
timeStart = new Date().getTime();
for(idx = 0; idx < arrLen; idx++ ) { z[idx] += x[idx] * y[idx] }
stopTimer(timeStart, arrLen, " c+=a*b ");
timeStart = new Date().getTime();
stopTimer(timeStart, arrLen, " sum ");
}
================================================
FILE: Examples/STREAMS/streams_fork_join.js
================================================
/*-----------------------------------------------------------------------------+
| Extended Memory Semantics (EMS) Version 1.0.0 |
| Synthetic Semantics http://www.synsem.com/ mogill@synsem.com |
+-----------------------------------------------------------------------------+
| Copyright (c) 2011-2014, Synthetic Semantics LLC. All rights reserved. |
| Copyright (c) 2015-2016, Jace A Mogill. All rights reserved. |
| |
| Redistribution and use in source and binary forms, with or without |
| modification, are permitted provided that the following conditions are met: |
| * Redistributions of source code must retain the above copyright |
| notice, this list of conditions and the following disclaimer. |
| * Redistributions in binary form must reproduce the above copyright |
| notice, this list of conditions and the following disclaimer in the |
| documentation and/or other materials provided with the distribution. |
| * Neither the name of the Synthetic Semantics nor the names of its |
| contributors may be used to endorse or promote products derived |
| from this software without specific prior written permission. |
| |
| THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS |
| "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT |
| LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR |
| A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL SYNTHETIC |
| SEMANTICS LLC BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, |
| EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, |
| PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR |
| PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF |
| LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING |
| NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS |
| SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. |
| |
+-----------------------------------------------------------------------------*/
//
// Based on John D. McCalpin's (Dr. Bandwdith) STREAMS benchmark:
// https://www.cs.virginia.edu/stream/
//
// Also see the Bulk Synchronous Parallel implementation in the Examples directory
// ===============================================================================
'use strict';
var ems = require('ems')(parseInt(process.argv[2]), true, 'fj');
ems.parallel( function() {
var arrLen = 1000000;
var array_a = ems.new(arrLen);
var array_b = ems.new(arrLen);
var array_c = ems.new(arrLen);
ems.parForEach(0, arrLen, function(idx) { array_a.write(idx, idx)});
ems.parForEach(0, arrLen, function(idx) { array_a.writeXE(idx, idx)});
ems.parForEach(0, arrLen, function(idx) { array_a.writeXF(idx, idx)});
ems.parForEach(0, arrLen, function(idx) { array_a.read(idx, idx)});
ems.parForEach(0, arrLen, function(idx) { array_a.readFF(idx, idx)});
ems.parForEach(0, arrLen, function(idx) { array_a.readFE(idx, idx)});
ems.parForEach(0, arrLen, function(idx) { array_a.writeEF(idx, idx)});
ems.parForEach(0, arrLen, function(idx) { array_b.writeXF(idx, array_a.readFF(idx))});
ems.parForEach(0, arrLen, function(idx) { array_c.writeXF(idx, array_a.readFF(idx) * array_b.readFF(idx))});
ems.parForEach(0, arrLen, function(idx) { array_c.writeXF(idx, array_c.readFF(idx) + (array_a.readFF(idx) * array_b.readFF(idx)))});
} );
// Fork-Join tasks must exit themselves, otherwise they will wait indefinitely for more work
process.exit(0);
================================================
FILE: Examples/WebServer/README.md
================================================
# Parallelism Generating a Single HTTP Response
Using Fork-Join (FJ) parallelism a conventional
HTTP server can use multiple cores to generate
a single a response by forking threads
when a request is received, generating the response in
parallel, joining all the threads when the work is complete,
and then returning the response from the original single thread.
HTTP applications frequently require state to persist between requests,
this example uses the URL path as a session ID that can be used to
persist state between requests. Each request itself may include
one or more fork-join parallel regions.
## Parallel Web Server Architecture
This example combines Node.js' built-in HTTP server module
with EMS for shared memory and task level parallelism.
When the server is started it allocates a globally shared
EMS shared memory (`serverState.ems`) that exists until the server exits.
The global shared memory is used for diagnostic counters like the
number of queries as well as key-value pairs that can be read
or written from a parallel or serial region of any request or session.
## REST API
A valid request URL consists of path naming the session ID
and a query that specifies if the session is new/restarting, or
is a continuation of a previous session.
__REST API:__ `/sessionID?&`
- `sessionID` - An unique session identifier. The shared memory
persistent file will be named `session.sessionID.ems` and stored
in the current working directory.
- `?` - A `new` session will remove an existing
session-private shared memory if present, an `old` session
will attach to a previously created shared memory that was
persisted to disk.
- `?` - Release or preserve, respectively, the
session's shared memory at the end of the request.
## Fork-Join Parallelism
A process using Fork-Join parallelism starts single-threaded,
creating parallel regions when needed, then returning to
single-threaded execution when all the threads in the
parallel region have completed execution.
EMS amortizes the cost of starting new processes by using a thread pool
where idle threads can quickly be restarted when they are needed again.
By way of comparison, all the processes in Bulk Synchronous Parallelism
are always executing, barriers can be used to synchronize execution.
Fork-Join Parallelism
Bulk Synchronous Parallelism
## Web Server Process
The web server is a conventional single-threaded based on Node.js'
built-in HTTP module. As part of the single-threaded initialization
EMS is initialized for Fork-Join parallelism and a shared memory
is created for the processes to share diagnostic counters and other
common state. This shared memory lasts for the life of the web
server process and does not persist between invocations of the server.
When a request is received, the URL is parsed, then the server enters
a parallel region. From the parallel region each process creates or
attaches to shared memory created
## Example
Start the server:
`node webServer.js [# of processes, default=8]`
From another terminal or browser, issue requests to the server.
A parallel region is created for each request received by the web server,
handling the single request with multiple processes.
Different variables in the response are used to illustrate
different types of data updates.
- `requestN` - Every request is given an unique ID
- `sessionData` - Each process records it's "work" in the private EMS
memory allocated for the session (`foo` or `bar`).
The results in the form
`(Req:/)`, the processes may complete
in any order, but each request is handled sequentially.
- `bignum` - The sum of a loop incrementing a counter in session
private EMS memory
- `randomSum` - The sum of random numbers generated by all the processes,
stored in the shared global EMS memory
Following the join at the end of the parallel region,
single-threaded execution continues, copying the different
variables into a response object
```
dunlin> curl 'http://localhost:8080/foo?new&keep' # Create a new persistent memory for session "foo"
{"requestN":0,"sessionData":"0(Req:0/2) (Req:0/7) (Req:0/1) (Req:0/4) (Req:0/5) (Req:0/3) (Req:0/0) (Req:0/6) ","bignum":100000,"randomSum":6103}
dunlin> curl 'http://localhost:8080/foo?old&keep' # Modify the existing "foo" memory
{"requestN":1,"sessionData":"0(Req:0/2) (Req:0/7) (Req:0/1) (Req:0/4) (Req:0/5) (Req:0/3) (Req:0/0) (Req:0/6) (Req:1/0) (Req:1/4) (Req:1/1) (Req:1/3) (Req:1/5) (Req:1/7) (Req:1/6) (Req:1/2) ","bignum":200000,"randomSum":16670}
dunlin> curl 'http://localhost:8080/bar?old' # Error because "bar" does not yet exist, 'keep' implied
ERROR: Requested "old" shared memory bar does not already exist.
dunlin> curl 'http://localhost:8080/bar?new&free' # Create and free session memory "bar"
{"requestN":3,"sessionData":"0(Req:3/4) (Req:3/7) (Req:3/5) (Req:3/3) (Req:3/0) (Req:3/6) (Req:3/2) (Req:3/1) ","bignum":100000,"randomSum":25530}
dunlin> curl 'http://localhost:8080/foo?old&keep' # Modify the existing "foo" memory again
{"requestN":4,"sessionData":"0(Req:0/2) (Req:0/7) (Req:0/1) (Req:0/4) (Req:0/5) (Req:0/3) (Req:0/0) (Req:0/6) (Req:1/0) (Req:1/4) (Req:1/1) (Req:1/3) (Req:1/5) (Req:1/7) (Req:1/6) (Req:1/2) (Req:4/0) (Req:4/5) (Req:4/2) (Req:4/7) (Req:4/6) (Req:4/1) (Req:4/3) (Req:4/4) ","bignum":300000,"randomSum":30114}
dunlin> curl 'http://localhost:8080/foo?new&free' # Modify the existing "foo" memory again, this time freeing it
{"requestN":5,"sessionData":"0(Req:5/4) (Req:5/7) (Req:5/5) (Req:5/0) (Req:5/2) (Req:5/6) (Req:5/1) (Req:5/3) ","bignum":100000,"randomSum":36887}
dunlin> curl 'http://localhost:8080/foo?old&free' # The memory "foo" has been released
ERROR: Requested "old" shared memory foo does not already exist.
```
###### Copyright (C)2017 Jace A Mogill - All Rights Reserved.
================================================
FILE: Examples/WebServer/webServer.js
================================================
/*-----------------------------------------------------------------------------+
| Extended Memory Semantics (EMS) Version 1.4.5 |
| Synthetic Semantics http://www.synsem.com/ mogill@synsem.com |
+-----------------------------------------------------------------------------+
| Copyright (c) 2011-2014, Synthetic Semantics LLC. All rights reserved. |
| Copyright (c) 2015-2017, Jace A Mogill. All rights reserved. |
| |
| Redistribution and use in source and binary forms, with or without |
| modification, are permitted provided that the following conditions are met: |
| * Redistributions of source code must retain the above copyright |
| notice, this list of conditions and the following disclaimer. |
| * Redistributions in binary form must reproduce the above copyright |
| notice, this list of conditions and the following disclaimer in the |
| documentation and/or other materials provided with the distribution. |
| * Neither the name of the Synthetic Semantics nor the names of its |
| contributors may be used to endorse or promote products derived |
| from this software without specific prior written permission. |
| |
| THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS |
| "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT |
| LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR |
| A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL SYNTHETIC |
| SEMANTICS LLC BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, |
| EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, |
| PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR |
| PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF |
| LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING |
| NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS |
| SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. |
| |
+-----------------------------------------------------------------------------*/
'use strict';
// Number of processes to use in parallel regions
var nProcs = process.argv[2] || 8;
// Initialize EMS for FJ parallelism
var ems = require('ems')(parseInt(nProcs), true, 'fj');
// Load modules for dealing with HTTP
var http = require('http');
// Load modules for parsing a request URL
var url = require('url');
// Port number to which the server listens
var port = 8080;
// Global, persistent, shared memory that is visible from any process during any HTTP request callback
var globallySharedData;
/**
* Fork into a parallel region to initialize each parallel processes
* by attaching the shared server status and loading 'require'd modules
*/
ems.parallel(function () {
globallySharedData = ems.new({
dimensions: [1000],
heapSize: [10000],
useExisting: false,
useMap: true,
doDataFill: true,
dataFill: 0,
filename: 'serverState.ems'
});
});
// Shared memory that last for a session, global so it is scoped to last more more than one request
var sessionSharedData;
/**
* When an HTTP request arrives this callback is executed.
* From within that callback, parallel processes are forked
* to work on the response, then join together back to the original
* callback thread from where the response is issued.
* @param request - Request object with URL
* @param response - Object used respond to the request
*/
function handleRequest(request, response) {
// Increment (Fetch And Add) the number of requests counter
var requestN = globallySharedData.faa('nRequests', 1);
// Request URL parsed into UTF8
var parsedRequest = url.parse(request.url, true);
// Session ID part of the request path
var key = parsedRequest.pathname.split(/\//g)[1];
if(!key) {
response.end('ERROR: Incorrectly formatted or missing session unique ID (' + parsedRequest.pathname + ')');
} else {
// Fork a parallel region
ems.parallel(parsedRequest, requestN, key, (parsedRequest, requestN, key) => {
/// EMS options for the session's shared memory
var sessionData = {
dimensions: [100], // Up to 100 keys in this memory
heapSize: [100000], // At least 100KB of memory
useMap: true, // Index using any type of key, not linear integer indexing
filename: 'session.' + key + '.ems', // Filename of persistent memory image
};
// Set EMS memory options to create a new or attach to an existing persistent EMS memory
if (parsedRequest.query.new === "") {
sessionData.dataFill = 0;
sessionData.doDataFill = true;
sessionData.useExisting = false;
sessionData.doSetFEtags = true;
sessionData.setFEtags = 'full';
} else if (parsedRequest.query.old === "") {
sessionData.useExisting = true;
} else {
// Send only one response, but all tasks exit the parallel region
if (ems.myID === 0) {
response.end('ERROR: Invalid new/old state (' + JSON.stringify(parsedRequest.query) + ') for private shared memory');
}
return;
}
// Create or attach to existing EMS memory
sessionSharedData = ems.new(sessionData);
if (!sessionSharedData) {
// Send only one response, but all tasks exit the parallel region
if (ems.myID === 0) {
response.end('ERROR: Requested "old" shared memory ' + key + ' does not already exist.');
}
return;
}
// Each task generates it's own random number and performs all the iterations.
var sum = 0;
for(var idx = 0; idx < Math.floor(Math.random() * 1000); idx += 1) {
sum += idx;
}
// All processes accumulate (Fetch And Add) directly into globally shared counter
globallySharedData.faa('sum', sum);
// Distribute (ie: map) the iterations of a loop across all the processes
ems.parForEach(0, 100000, function(idx) {
sessionSharedData.faa('bignum', 1); // Accumulate into a per-session counter
});
// Each process appends it's results to the session's EMS memory using Fetch And Add.
// Note that the memory was initialized with 0 to which this text is appended
sessionSharedData.faa('results', "(Req:" + requestN + "/" + ems.myID + ") ");
});
// All parallel processes have returned before the serial region resumes here
/// Construct a return object containing data from both the globally shared memory
/// and session-private data.
var resp = { 'requestN' : requestN };
// Parallel region may have exited due to error, in which case there are no results to return
if (sessionSharedData) {
resp.sessionData = sessionSharedData.readFF('results');
resp.bignum = sessionSharedData.readFF('bignum');
resp.randomSum = globallySharedData.readFF('sum');
response.end(JSON.stringify(resp));
if (parsedRequest.query.free === "") {
sessionSharedData.destroy(true);
}
} else {
response.end('ERROR: Parallel region exited without creating/accessing session shared memory');
}
}
}
// Create the Web server
http.createServer(handleRequest).listen(port, function () {
console.log(
"REST API: /UID?[old|new]&[keep|free]\n" +
" /UID Unique session identifier\n" +
" ?[old|new] A 'new' session will remove and create a new named private shared memory\n" +
" ?[keep|free] Release or preserve, respectively, the named private shared memory\n" +
"\nExamples:\n" +
" curl http://localhost:8080/foo?new&keep # Create a new persistent memory for session \"foo\"\n" +
" curl http://localhost:8080/foo?old&keep # Modify the existing \"foo\" memory\n" +
" curl http://localhost:8080/bar?old # Error because \"bar\" does not yet exist\n" +
" curl http://localhost:8080/bar?new&free # Create and free session memory \"bar\"\n" +
"\nServer listening on: http://localhost:" + port + "\n"
);
});
================================================
FILE: Examples/concurrent_Q_and_TM.js
================================================
/*-----------------------------------------------------------------------------+
| Extended Memory Semantics (EMS) Version 1.0.0 |
| Synthetic Semantics http://www.synsem.com/ mogill@synsem.com |
+-----------------------------------------------------------------------------+
| Copyright (c) 2011-2014, Synthetic Semantics LLC. All rights reserved. |
| Copyright (c) 2015-2016, Jace A Mogill. All rights reserved. |
| |
| Redistribution and use in source and binary forms, with or without |
| modification, are permitted provided that the following conditions are met: |
| * Redistributions of source code must retain the above copyright |
| notice, this list of conditions and the following disclaimer. |
| * Redistributions in binary form must reproduce the above copyright |
| notice, this list of conditions and the following disclaimer in the |
| documentation and/or other materials provided with the distribution. |
| * Neither the name of the Synthetic Semantics nor the names of its |
| contributors may be used to endorse or promote products derived |
| from this software without specific prior written permission. |
| |
| THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS |
| "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT |
| LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR |
| A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL SYNTHETIC |
| SEMANTICS LLC BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, |
| EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, |
| PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR |
| PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF |
| LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING |
| NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS |
| SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. |
| |
+-----------------------------------------------------------------------------+
| Program Description: |
| |
| Sequentially enqueue transactions from thread 0 while all other |
| threads consume them. When all the operations have been queued |
| thread 0 also begins dequeing work and processing transactions. |
| |
| Unlike the workQ_and_TM.js example, randomInRange() is only called |
| from thread 0 so it is defined only in the main thread's context. |
| |
+-----------------------------------------------------------------------------*/
'use strict';
// Initialize EMS to use the fork-join execution model
var ems = require('ems')(parseInt(process.argv[2]), true, 'fj');
var assert = require('assert');
var workQ;
var totalNops;
var checkNops;
var arrLen;
var heapSize;
var nTransactions;
var nTables;
var maxNops;
var tables;
//---------------------------------------------------------------------------
// Generate a random integer within a range (inclusive) from 'low' to 'high'
//
function randomInRange(low, high) {
return (Math.floor((Math.random() * (high - low)) + low));
}
//-------------------------------------------------------------------
// Timer functions
function timerStart() {
return new Date().getTime();
}
function timerStop(timer, nOps, label, myID) {
function fmtNumber(n) {
var s = ' ' + n.toString().replace(/\B(?=(\d{3})+(?!\d))/g, ",");
if (n < 1) return n;
else {
return s.substr(s.length - 15, s.length);
}
}
var now = new Date().getTime();
var opsPerSec = (nOps * 1000000) / ((now - timer) * 1000);
if (typeof myID === undefined || myID === 0) {
console.log(fmtNumber(nOps) + label + fmtNumber(Math.floor(opsPerSec).toString()) + " ops/sec");
}
}
//---------------------------------------------------------------------------
// Initialize shared data: global scalars, EMS buffers for statistics
// and checksums, an EMS array to be used as a work queue, and many
// tables to perform transactions on.
//
function initializeSharedData() {
arrLen = 1000000; // Maximum number of elements the EMS array can contain
heapSize = 100000; // Amount of persistent memory to reserve for transactions
nTransactions = 1000000; // Number of transactions to perform in the experiment
nTables = 6; // Number of EMS arrays to perform transactions across
maxNops = 5; // Maximum number of EMS arrays to update during a transaction
tables = [];
totalNops = ems.new(2);
checkNops = ems.new(1);
//---------------------------------------------------------------------------
// The queue of transactions to perform
// [ table#, index, read-only ]
workQ = ems.new({
dimensions: [nTransactions + ems.nThreads],
heapSize: nTransactions * 200,
useExisting: false,
setFEtags: 'empty'
});
//---------------------------------------------------------------------------
// Create all the tables
for (var tableN = 0; tableN < nTables; tableN++) {
tables[tableN] = ems.new({
dimensions: [arrLen],
heapSize: 0,
useExisting: false,
filename: '/tmp/EMS_tm' + tableN,
doDataFill: true,
dataFill: 0,
setFEtags: 'full'
});
}
}
//---------------------------------------------------------------------------
// Create 'nTransactions' many transactions, each having a random number (up
// to maxNops) randomly chosen values, some of which are read-only.
// Because tables and elements are chosen at random, it is necessary to
// remove duplicate elements in a single transaction, to prevent deadlocks.
//
// Transactions are pushed onto a shared queue
function generateTransactions() {
//---------------------------------------------------------------------------
// Generate operations involving random elements in random EMS arrays
// and enqueue them on the work queue
for (var transN = 0; transN < nTransactions; transN++) {
var ops = [];
var nOps = randomInRange(1, maxNops);
// var indexes = [];
for (var opN = 0; opN < nOps; opN++) {
var tableN = randomInRange(0, nTables);
var idx = randomInRange(0, arrLen);
if (transN % 2 == 0 || opN % 3 > 0) {
ops.push([tableN, idx, true]);
} else {
ops.push([tableN, idx]);
}
}
// De-duplicate operands in a single transaction as that would deadlock
var indicies = [];
var uids = [];
for (var i = 0; i < ops.length; ++i) {
indicies[i] = i;
uids[i] = (ops[i][0] * 1000000000000) + ops[i][1];
}
var uniq = [];
for (opN = 0; opN < ops.length; opN++) {
var isDupl = false;
for (var checkN = 0; checkN < ops.length; checkN++) {
if (opN != checkN && uids[opN] == uids[checkN]) {
isDupl = true;
break;
}
}
if (!isDupl) {
uniq.push(ops[opN]);
}
}
workQ.enqueue(JSON.stringify(uniq));
}
}
//------------------------------------------------------------------
// Simultaneously generate and consume transactions
// Thread 0 enqueues them while the rest dequeue transactions.
// When thread 0 finishes creating new work, it also begins to
// dequeue transactions from the workQ.
//
// If there is nothing on the queue, keep checking until the "DONE"
// message is dequeued.
//
function performTransactions() {
//------------------------------------------------------------------
// Generate the transactions concurrently with their consumption
if (ems.myID == 0) {
var startTime = timerStart();
generateTransactions();
// After all the work has been enqueued, add DONE semaphores to the
// end of the queue so they are processed only after all the work
// has been issued. Each thread enqueues one event and can only
// consume one before exiting.
for (var taskN = 0; taskN < ems.nThreads; taskN++) {
workQ.enqueue("DONE");
}
timerStop(startTime, nTransactions, " transactions enqueued ", ems.myID);
}
//------------------------------------------------------------------
// Process the transactions
var rwNops = 0;
var readNops = 0;
while (true) {
var str = workQ.dequeue();
if (str !== undefined) {
if (str === "DONE") {
break
} else {
var ops = JSON.parse(str);
for (var opN = 0; opN < ops.length; opN++) {
ops[opN][0] = tables[ops[opN][0]];
}
var transaction = ems.tmStart(ops);
ops.forEach(function (op) {
var tmp = op[0].read(op[1]);
if (op[2] != true) {
rwNops++;
op[0].write(op[1], tmp + 1);
} else {
readNops++;
}
});
ems.tmEnd(transaction, true);
}
} else {
// Queue was empty, but have not yet seen DONE event, so
// keep trying to dequeue work
}
}
totalNops.faa(0, rwNops);
totalNops.faa(1, readNops);
}
//=======================================================================
// B E G I N P R O G R A M M A I N
//-------------------------------------------------------
// Initialize the shared data
var startTime = timerStart();
ems.parallel(initializeSharedData);
timerStop(startTime, nTables, " tables initialized ", ems.myID);
// Data to be initialized only once
totalNops.writeXF(0, 0);
totalNops.writeXF(1, 0);
checkNops.writeXF(0, 0);
// Perform all the transactions
startTime = timerStart();
ems.parallel(performTransactions);
timerStop(startTime, nTransactions, " transactions performed", ems.myID);
timerStop(startTime, totalNops.readFF(0), " table updates ", ems.myID);
timerStop(startTime, totalNops.readFF(0) + totalNops.readFF(1), " elements referenced ", ems.myID);
//------------------------------------------------------------------
// Sum all the values in the tables to account for all the transactions
startTime = timerStart();
ems.parallel(function () {
ems.parForEach(0, nTables, function (tableN) {
var localSum = 0;
for (var idx = 0; idx < arrLen; idx++) {
localSum += tables[tableN].read(idx);
}
checkNops.faa(0, localSum);
});
});
timerStop(startTime, nTables * arrLen, " elements checked ", ems.myID);
assert(checkNops.readFF(0) == totalNops.readFF(0),
"Error in final sum = " + checkNops.readFF(0) + " should be=" + totalNops.readFF(0));
console.log("Results are correct");
// One liner to get all processes to exit
ems.parallel(function () { process.exit(0) });
================================================
FILE: Examples/wordCount.js
================================================
/*-----------------------------------------------------------------------------+
| Extended Memory Semantics (EMS) Version 1.4.0 |
| Synthetic Semantics http://www.synsem.com/ mogill@synsem.com |
+-----------------------------------------------------------------------------+
| Copyright (c) 2011-2014, Synthetic Semantics LLC. All rights reserved. |
| Copyright (c) 2015-2016, Jace A Mogill. All rights reserved. |
| |
| Redistribution and use in source and binary forms, with or without |
| modification, are permitted provided that the following conditions are met: |
| * Redistributions of source code must retain the above copyright |
| notice, this list of conditions and the following disclaimer. |
| * Redistributions in binary form must reproduce the above copyright |
| notice, this list of conditions and the following disclaimer in the |
| documentation and/or other materials provided with the distribution. |
| * Neither the name of the Synthetic Semantics nor the names of its |
| contributors may be used to endorse or promote products derived |
| from this software without specific prior written permission. |
| |
| THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS |
| "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT |
| LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR |
| A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL SYNTHETIC |
| SEMANTICS LLC BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, |
| EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, |
| PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR |
| PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF |
| LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING |
| NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS |
| SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. |
| |
+-----------------------------------------------------------------------------*/
'use strict';
var ems = require('ems')(parseInt(process.argv[2]));
var fs = require('fs');
//-------------------------------------------------------------------
// Allocate the word count dictionary
var maxNKeys = 100000000;
var wordCounts = ems.new({
dimensions: [maxNKeys], // Maximum # of different keys the array can store
heapSize: maxNKeys * 10, // 10 bytes of storage per key, used for key (dictionary word) itself
useMap: true, // Use a key-index mapping, not integer indexes
setFEtags: 'full', // Initial full/empty state of array elements
doDataFill: true, // Initialize data values
dataFill: 0 // Initial value of new keys
});
//-------------------------------------------------------------------
// Use a key-value mapped array to store execution statistics
var nStats = 200;
var stats = ems.new({
dimensions: [nStats], // Maximum number of stats that can be stored
heapSize: nStats * 200,// Space for keys, data values, and sorting
useMap: true, // Use a key-index mapping, not integer indexes
setFEtags: 'full', // Initial full/empty state of array elements
doDataFill: true, // Initialize data values
dataFill: 0 // Initial value of new keys
});
// ===============================================================================
// Perform Word Count
//
// Use the word as a key to the EMS array, the value is the count of the
// number of times the word has been encountered. This is incremented
// atomically.
//
// Bookkeeping for statistics is performed by keeping track of the total number
// of words and bytes processed in the "stats" EMS array.
//
var doc_path = '/path/to/your/document/collection/';
if (!process.argv[3]) {
console.log("usage: wordCount <# processes> /path/to/your/document/collection/");
return -1;
} else {
doc_path = process.argv[3];
}
var file_list = fs.readdirSync(doc_path);
var splitPattern = new RegExp(/[ \n,\.\\/_\-\<\>:\;\!\@\#\$\%\&\*\(\)=\[\]|\"\'\{\}\?\—]/);
// Loop over the files in parallel, counting words
var startTime = Date.now();
ems.parForEach(0, file_list.length, function (fileNum) {
try {
var text = fs.readFileSync(doc_path + file_list[fileNum], 'utf8', "r");
var words = text.replace(/[\n\r]/g, ' ').toLowerCase().split(splitPattern);
// Iterate over all the words in the document
words.forEach(function (word) {
// Ignore words over 15 characters long as non-English
if (word.length < 15 && word.length > 0) {
// Atomically increment the count of times this word was seen
var count = wordCounts.faa(word, 1);
}
});
// Accumulate some statistics: word and data counters, print progress
var nWords_read = stats.faa('nWords', words.length);
var nBytes_read = stats.faa('nBytesRead', text.length);
var now = Date.now();
if (ems.myID === 0 && fileNum % 10 === 0) {
console.log("Average Words/sec=" + Math.floor((1000 * nWords_read) / (now - startTime)) +
" MBytes/sec=" + Math.floor((100000 * nBytes_read) / ((now - startTime) * (1 << 20)))/100);
}
}
catch (Err) {
ems.diag("This is not a text file:" + file_list[fileNum]);
}
});
// ===============================================================================
// Output the Most Frequently Occurring Words
//
// First perform many sequential insertion sorts in parallel,
// then serially perform a merge sort of all the partial lists.
//
// Start by printing the total amount of data processed during word counting
ems.master(function() {
console.log("Totals: ", stats.read('nWords'), " words parsed, ",
stats.read('nBytesRead'), "bytes read.");
console.log("Starting sort....");
});
// Divide the array across all the processes, each process keeps track
// of the "local_sort_len" most frequent word it encounters.
var local_sort_len = Math.max(10, process.argv[2]);
var biggest_counts = new Array(local_sort_len).fill({"key": 0, "count": 0});
ems.parForEach(0, maxNKeys, function (keyN) {
if (keyN % (maxNKeys / 10) === 0) { ems.diag("Sorting key ID " + keyN); }
var key = wordCounts.index2key(keyN);
if (key) {
// Perform an insertion sort of the new key into the biggest_counts
// list, deleting the last (smallest) element to preserve length.
var keyCount = wordCounts.read(key);
var idx = local_sort_len - 1;
while (idx >= 0 && biggest_counts[idx].count < keyCount) {
idx -= 1;
}
var newBiggest = {"key": key, "count": keyCount};
if (idx < 0) {
biggest_counts = [newBiggest].concat(biggest_counts.slice(0, biggest_counts.length - 1));
} else if (idx < local_sort_len) {
var left = biggest_counts.slice(0, idx + 1);
var right = biggest_counts.slice(idx + 1);
biggest_counts = left.concat([newBiggest].concat(right)).slice(0, -1);
}
}
});
// Concatenate all the partial (one per process) lists into one list
stats.writeXF('most_frequent', []);
ems.barrier();
stats.writeEF('most_frequent', stats.readFE('most_frequent').concat(biggest_counts));
ems.barrier();
// Sort & print the list of words, only one process is needed
ems.master(function() {
biggest_counts = stats.readFF('most_frequent');
// Sort the list by word frequency
biggest_counts.sort(function (a, b) {
return b.count - a.count;
});
// Print only the first "local_sort_len" items -- assume the worst case
// of all the largest counts are discovered by a single process, the next
// largest after "local_sort_len" is no longer on any list.
console.log("Most frequently appearing terms:");
for (var index = 0; index < local_sort_len; index += 1) {
console.log(index + ': ' + biggest_counts[index].key + " " + biggest_counts[index].count);
}
});
================================================
FILE: Examples/workQ_and_TM.js
================================================
/*-----------------------------------------------------------------------------+
| Extended Memory Semantics (EMS) Version 1.0.0 |
| Synthetic Semantics http://www.synsem.com/ mogill@synsem.com |
+-----------------------------------------------------------------------------+
| Copyright (c) 2011-2014, Synthetic Semantics LLC. All rights reserved. |
| Copyright (c) 2015-2016, Jace A Mogill. All rights reserved. |
| |
| Redistribution and use in source and binary forms, with or without |
| modification, are permitted provided that the following conditions are met: |
| * Redistributions of source code must retain the above copyright |
| notice, this list of conditions and the following disclaimer. |
| * Redistributions in binary form must reproduce the above copyright |
| notice, this list of conditions and the following disclaimer in the |
| documentation and/or other materials provided with the distribution. |
| * Neither the name of the Synthetic Semantics nor the names of its |
| contributors may be used to endorse or promote products derived |
| from this software without specific prior written permission. |
| |
| THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS |
| "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT |
| LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR |
| A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL SYNTHETIC |
| SEMANTICS LLC BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, |
| EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, |
| PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR |
| PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF |
| LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING |
| NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS |
| SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. |
| |
+-----------------------------------------------------------------------------+
| Program Description: |
| |
| Enqueue the transactions from a parallel loop. |
| When all the operations have been queued they all begin |
| processing transactions. |
| |
| Unlike the concurrent_Q_and_TM.js example, randomInRange() is called |
| from every thread, so it is declared from every thread. |
| |
+-----------------------------------------------------------------------------*/
'use strict';
var ems = require('ems')(parseInt(process.argv[2]), true, 'fj');
var assert = require('assert');
var workQ;
var totalNops;
var checkNops;
var arrLen;
var heapSize;
var nTransactions;
var nTables;
var maxNops;
var tables;
//-------------------------------------------------------------------
// Timer functions
function timerStart() {
return new Date().getTime();
}
function timerStop(timer, nOps, label, myID) {
function fmtNumber(n) {
var s = ' ' + n.toString().replace(/\B(?=(\d{3})+(?!\d))/g, ",");
if (n < 1) return n;
else {
return s.substr(s.length - 15, s.length);
}
}
var now = new Date().getTime();
var opsPerSec = (nOps * 1000000) / ((now - timer) * 1000);
if (typeof myID === undefined || myID === 0) {
console.log(fmtNumber(nOps) + label + fmtNumber(Math.floor(opsPerSec).toString()) + " ops/sec");
}
}
//---------------------------------------------------------------------------
// Initialize shared data: global scalars, EMS buffers for statistics
// and checksums, an EMS array to be used as a work queue, and many
// tables to perform transactions on.
//
function initializeSharedData() {
arrLen = 1000000; // Maximum number of elements the EMS array can contain
heapSize = 100000; // Amount of persistent memory to reserve for transactions
nTransactions = 1000000; // Number of transactions to perform in the experiment
nTables = 6; // Number of EMS arrays to perform transactions across
maxNops = 5; // Maximum number of EMS arrays to update during a transaction
tables = [];
totalNops = ems.new(2);
checkNops = ems.new(1);
//---------------------------------------------------------------------------
// The queue of transactions to perform
// [ table#, index, read-only ]
workQ = ems.new({
dimensions: [nTransactions + ems.nThreads],
heapSize: nTransactions * 200,
useExisting: false,
dataFill: 0,
doDataFill: true,
setFEtags: 'empty'
});
//---------------------------------------------------------------------------
// Create all the tables
for (var tableN = 0; tableN < nTables; tableN++) {
tables[tableN] = ems.new({
dimensions: [arrLen],
heapSize: 0,
useExisting: false,
filename: '/tmp/EMS_tm' + tableN,
dataFill: 0,
doDataFill: true,
setFEtags: 'full'
});
}
}
//---------------------------------------------------------------------------
// Create 'nTransactions' many transactions, each having a random number (up
// to maxNops) randomly chosen values, some of which are read-only.
// Because tables and elements are chosen at random, it is necessary to
// remove duplicate elements in a single transaction, to prevent deadlocks.
//
// Transactions are pushed onto a shared queue
function generateTransactions() {
//---------------------------------------------------------------------------
// Generate a random integer within a range (inclusive) from 'low' to 'high'
var randomInRange = function (low, high) {
return ( Math.floor((Math.random() * (high - low)) + low) );
};
//---------------------------------------------------------------------------
// Generate operations involving random elements in random EMS arrays
// and enqueue them on the work queue
ems.parForEach(0, nTransactions, function (transN) {
var ops = [];
var nOps = randomInRange(1, maxNops);
for (var opN = 0; opN < nOps; opN++) {
var tableN = randomInRange(0, nTables);
var idx = randomInRange(0, arrLen);
if (transN % 2 == 0 || opN % 3 > 0) {
ops.push([tableN, idx, true]);
}
else {
ops.push([tableN, idx]);
}
}
// De-duplicate operations in a transaction which would deadlock
var indicies = [];
var uids = [];
for (var i = 0; i < ops.length; ++i) {
indicies[i] = i;
uids[i] = (ops[i][0] * 1000000000000) + ops[i][1];
}
var uniq = [];
for (opN = 0; opN < ops.length; opN++) {
var isDupl = false;
for (var checkN = 0; checkN < ops.length; checkN++) {
if (opN != checkN && uids[opN] == uids[checkN]) {
isDupl = true;
break;
}
}
if (!isDupl) {
uniq.push(ops[opN]);
}
}
workQ.enqueue(uniq);
});
// After all the work has been enqueued, add DONE semaphores to the
// end of the queue so they are processed only after all the work
// has been issued. Each thread enqueues one event and can only
// consume one before exiting.
workQ.enqueue("DONE")
}
//------------------------------------------------------------------
// Consume transactions. If there is nothing on the queue,
// keep checking until the "DONE" message is dequeued.
//
function performTransactions() {
var rwNops = 0;
var readNops = 0;
while (true) {
var ops = workQ.dequeue();
if (ops !== undefined) {
if (ops === "DONE") {
break;
} else {
for (var opN = 0; opN < ops.length; opN++) {
ops[opN][0] = tables[ops[opN][0]];
}
var transaction = ems.tmStart(ops);
ops.forEach(function (op) {
var tmp = op[0].read(op[1]);
if (op[2] != true) {
rwNops++;
op[0].write(op[1], tmp + 1);
} else {
readNops++;
}
});
ems.tmEnd(transaction, true);
}
}
}
totalNops.faa(0, rwNops);
totalNops.faa(1, readNops);
}
//------------------------------------------------------------------------
// Main program entry point
//
ems.parallel(initializeSharedData);
totalNops.writeXF(0, 0);
totalNops.writeXF(1, 0);
checkNops.writeXF(0, 0);
// Enqueue all the transaction
var startTime = timerStart();
ems.parallel(generateTransactions);
timerStop(startTime, nTransactions, " transactions enqueued ", ems.myID);
// Perform all the transactions
startTime = timerStart();
ems.parallel(performTransactions);
timerStop(startTime, nTransactions, " transactions performed", ems.myID);
timerStop(startTime, totalNops.readFF(0), " table updates ", ems.myID);
timerStop(startTime, totalNops.readFF(0) + totalNops.readFF(1), " elements referenced ", ems.myID);
// Validate the results by summing all the updates
startTime = timerStart();
ems.parallel(function () {
ems.parForEach(0, nTables, function (tableN) {
var localSum = 0;
for (var idx = 0; idx < arrLen; idx++) {
localSum += tables[tableN].read(idx);
}
checkNops.faa(0, localSum)
})
});
timerStop(startTime, nTables * arrLen, " elements checked ", ems.myID);
assert(checkNops.readFF(0) === totalNops.readFF(0),
"Error in final sum = " + checkNops.readFF(0) + " should be=" + totalNops.readFF(0));
console.log("Success: Got correct final sum = " + checkNops.readFF(0));
// No more work coming -- Exit on all processes
ems.parallel(function () { process.exit(0); });
================================================
FILE: LICENSE
================================================
/*-----------------------------------------------------------------------------+
| Extended Memory Semantics (EMS) Version 1.6.1 |
+-----------------------------------------------------------------------------+
| Copyright (c) 2011-2014, Synthetic Semantics LLC. All rights reserved. |
| Copyright (c) 2015-2020, Jace A Mogill. All rights reserved. |
| |
| Redistribution and use in source and binary forms, with or without |
| modification, are permitted provided that the following conditions are met: |
| * Redistributions of source code must retain the above copyright |
| notice, this list of conditions and the following disclaimer. |
| * Redistributions in binary form must reproduce the above copyright |
| notice, this list of conditions and the following disclaimer in the |
| documentation and/or other materials provided with the distribution. |
| * Neither the name of the Synthetic Semantics nor the names of its |
| contributors may be used to endorse or promote products derived |
| from this software without specific prior written permission. |
| |
| THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS |
| "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT |
| LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR |
| A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL SYNTHETIC |
| SEMANTICS LLC BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, |
| EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, |
| PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR |
| PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF |
| LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING |
| NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS |
| SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. |
| |
+-----------------------------------------------------------------------------*/
================================================
FILE: Makefile
================================================
#make
# *-----------------------------------------------------------------------------+
# | Extended Memory Semantics (EMS) Version 1.6.1 |
# | http://mogill.com/ jace@mogill.com |
# +-----------------------------------------------------------------------------+
# | Copyright (c) 2016-2020, Jace A Mogill. All rights reserved. |
# | |
# | Redistribution and use in source and binary forms, with or without |
# | modification, are permitted provided that the following conditions are met: |
# | * Redistributions of source code must retain the above copyright |
# | notice, this list of conditions and the following disclaimer. |
# | * Redistributions in binary form must reproduce the above copyright |
# | notice, this list of conditions and the following disclaimer in the |
# | documentation and/or other materials provided with the distribution. |
# | * Neither the name of the Synthetic Semantics nor the names of its |
# | contributors may be used to endorse or promote products derived |
# | from this software without specific prior written permission. |
# | |
# | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS |
# | "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT |
# | LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR |
# | A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL SYNTHETIC |
# | SEMANTICS LLC BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, |
# | EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, |
# | PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR |
# | PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF |
# | LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING |
# | NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS |
# | SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. |
# | |
# +-----------------------------------------------------------------------------*
all: py2 py3 node test help_notice
help:
@echo " Extended Memory Semantics -- Build Targets"
@echo "==========================================================="
@echo " all Build all targets, run all tests"
@echo " node Build only Node.js"
@echo " py Build both Python 2 and 3"
@echo " "
@echo " py[2|3] Build only Python2 or 3"
@echo " test Run both Node.js and Py tests"
@echo " test[_js|_py|_py2|_py3] Run only Node.js, or only Py tests, respectively"
@echo " clean Remove all files that can be regenerated"
@echo " clean[_js|_py|_py2|_py3] Remove Node.js or Py files that can be regenerated"
help_notice:
@echo "=== \"make help\" for list of targets"
test: test_js test_py
test_js: node
npm test
test_py: test_py2 test_py3
test_py3: py3
(cd Tests; python3 ./py_api.py)
test_py2: py2
(cd Tests; python ./py_api.py)
node: build/Release/ems.node
build/Release/ems.node:
(cd node_modules; /bin/rm -f ems)
npm install
(cd node_modules; ln -s ../ ./ems)
py: py2 py3
py3:
(cd Python; sudo rm -rf Python/build Python/ems.egg-info Python/dist; sudo python3 ./setup.py build --build-temp=./ install)
py2:
(cd Python; sudo rm -rf Python/build Python/ems.egg-info Python/dist; sudo python ./setup.py build --build-temp=./ install)
clean: clean_js clean_py3 clean_py2
clean_js:
$(RM) -rf build
clean_py3:
$(RM) -rf Python/build Python/py3ems/build /usr/local/lib/python*/dist-packages/*ems* ~/Library/Python/*/lib/python/site-packages/*ems* ~/Library/Python/*/lib/python/site-packages/__pycache__/*ems* /Library/Frameworks/Python.framework/Versions/*/lib/python*/site-packages/*ems*
clean_py2:
$(RM) -rf Python/build Python/py2ems/build /usr/local/lib/python*/dist-packages/*ems* ~/Library/Python/*/lib/python/site-packages/*ems* ~/Library/Python/*/lib/python/site-packages/__pycache__/*ems* /Library/Python/*/site-packages/*ems*
================================================
FILE: Python/README.md
================================================
# Python 2 and 3
The FFI differs slightly between Python 2 and 3, these are papered over
at runtime. There are, of course, the usual "print" shenanigans.
Using the "test_py" make target, both platforms are built and tested.
The user API does not change between Python 2 and 3.
================================================
FILE: Python/__init__.py
================================================
from ems import *
================================================
FILE: Python/ems.py
================================================
"""
+-----------------------------------------------------------------------------+
| Extended Memory Semantics (EMS) Version 1.5.0 |
| Synthetic Semantics http://www.synsem.com/ mogill@synsem.com |
+-----------------------------------------------------------------------------+
| Copyright (c) 2016-2017, Jace A Mogill. All rights reserved. |
| |
| Redistribution and use in source and binary forms, with or without |
| modification, are permitted provided that the following conditions are met: |
| * Redistributions of source code must retain the above copyright |
| notice, this list of conditions and the following disclaimer. |
| * Redistributions in binary form must reproduce the above copyright |
| notice, this list of conditions and the following disclaimer in the |
| documentation and/or other materials provided with the distribution. |
| * Neither the name of the Synthetic Semantics nor the names of its |
| contributors may be used to endorse or promote products derived |
| from this software without specific prior written permission. |
| |
| THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS |
| "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT |
| LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR |
| A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL SYNTHETIC |
| SEMANTICS LLC BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, |
| EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, |
| PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR |
| PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF |
| LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING |
| NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS |
| SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. |
| |
+-----------------------------------------------------------------------------+
"""
import sys
import os
import math
import json
import re
import subprocess
from multiprocessing import Process, Pipe
from cffi import FFI
import site
"""
sudo apt-get install python-pip
sudo apt-get install libffi-dev
sudo pip install cffi
sudo apt-get install python3-cffi
"""
ffi = FFI()
cpp_out = subprocess.check_output(["cpp", "../src/ems_proto.h"]).decode("utf-8")
prototypes = cpp_out.split("\n")
headerLines = []
for line in prototypes:
# Strip CPP directives and annotations
line = re.sub("^#.*$", "", line)
# Strip extern attribute
line = re.sub("extern \"C\" ", "", line)
if line is not "":
headerLines.append(line)
# Delcare the CFFI Headers
ffi.cdef('\n'.join(headerLines))
# Find the .so and load it
libems = None
package_paths = site.getsitepackages()
for package_path in package_paths:
try:
packages = os.listdir(package_path)
for package in packages:
if package == "libems" and libems is None:
libems_path = package_path + "/libems/"
files = os.listdir(libems_path)
for file in files:
if file[-3:] == ".so": # TODO: Guessing it's the only .so
fname = libems_path + file
libems = ffi.dlopen(fname)
break
except:
# print("This path does not exist:", package_path, "|", type (package_path))
pass
# Do not GC the EMS values until deleted
import weakref
global_weakkeydict = weakref.WeakKeyDictionary()
# class initialize(object):
# This enumeration is copied from ems.h
TYPE_INVALID = 0
TYPE_BOOLEAN = 1
TYPE_STRING = 2
TYPE_FLOAT = 3
TYPE_INTEGER = 4
TYPE_UNDEFINED = 5
TYPE_JSON = 6 # Catch-all for JSON arrays and Objects
TAG_ANY = 4 # Never stored, used for matching
TAG_RW_LOCK = 3
TAG_BUSY = 2
TAG_EMPTY = 1
TAG_FULL = 0
def emsThreadStub(conn, taskn):
"""Function that receives EMS fork-join functions, executes them,
and returns the results."""
sys.path.append("../Python/")
global ems
import ems
ems.initialize(taskn, True, 'fj', '/tmp/fj_main.ems')
print("STUB: This is the start of it", taskn)
ems.myID = taskn
ems.diag("Stub diag. Taskn=" + str(taskn) + " myid=" + str(ems.myID))
while True:
msg = conn.recv()
print("CONN RECV ON", taskn)
func = msg['func']
args = msg['args']
if func is None:
print("Farewell!!!!!!!! from ", taskn)
conn.close()
exit(0)
ems.diag("func=" + str(func) + " args=" + str(args))
# print("func=" + str(func) + " args=" + str(args))
retval = func(*args)
conn.send(str(retval))
# time.sleep(taskn * 0.01)
def initialize(nThreadsArg, pinThreadsArg=False, threadingType='bsp',
contextname='/EMS_MainDomain'):
"""EMS object initialization, invoked by the require statement"""
nThreadsArg = int(nThreadsArg)
global myID, libems, EMSmmapID, _regionN, pinThreads, domainName, inParallelContext, tasks, nThreads
if not nThreadsArg > 0:
print("EMS: Must declare number of nodes to use. Input:" + str(nThreadsArg))
return
myID = os.getenv("EMS_Subtask")
if myID is None:
myID = 0
else:
myID = int(myID)
_regionN = 0
nThreads = nThreadsArg
pinThreads = pinThreadsArg
threadingType = threadingType
domainName = contextname
c_contextname = _new_EMSval(contextname)
c_None = _new_EMSval(None)
c_contextname_char = ffi.cast('char *', c_contextname)
# All arguments are defined -- now do the EMS initialization
EMSmmapID = libems.EMSinitialize(0, 0, False,
c_contextname_char,
False, False, # 4-5
False, 0, # 6-7
c_None, # 8
False, TAG_FULL, myID, pinThreads, nThreads, 99)
# The master thread has completed initialization, other threads may now
# safely execute.
if threadingType == 'bsp':
inParallelContext = True
if myID == 0:
for taskN in range(1, nThreads):
py_ver = "python"
if sys.version_info[0] == 3:
py_ver += "3"
os.system('EMS_Subtask=' + str(taskN) + " " + py_ver + ' ./' + (' '.join(sys.argv)) + ' &')
elif threadingType == 'fj':
if myID == 0:
inParallelContext = False
tasks = []
for taskN in range(1, nThreads):
parent_conn, child_conn = Pipe()
p = Process(target=emsThreadStub, args=(child_conn, taskN), name="EMS" + str(taskN))
p.start()
print("Started job:", taskN)
tasks.append((p, parent_conn, child_conn))
else:
inParallelContext = True
elif threadingType == 'user':
inParallelContext = False
else:
print("EMS ERROR: Unknown threading model type:" + str(threadingType))
myID = -1
return
def _loop_chunk():
global myID, libems, EMSmmapID, _regionN, pinThreads, domainName, inParallelContext, tasks, nThreads
start = ffi.new("int32_t *", 1)
end = ffi.new("int32_t *", 1)
if not libems.EMSloopChunk(EMSmmapID, start, end):
diag("_loop_chunk: ERROR -- Not a valid block of iterations")
return False
return {'start': start[0], 'end': end[0]}
def diag(text):
"""Print a message to the console with a prefix indicating which thread is printing
"""
global myID, libems, EMSmmapID, _regionN, pinThreads, domainName, inParallelContext, tasks, nThreads
print("EMStask " + str(myID) + ": " + text)
def parallel(func, *kargs):
"""Co-Begin a FJ parallel region, executing the function 'func'"""
global myID, libems, EMSmmapID, _regionN, pinThreads, domainName, inParallelContext, tasks, nThreads
# print("EMSparallel: starting", len(tasks), tasks)
inParallelContext = True
taskN = 1
for proc, parent_conn, child_conn in tasks:
print("Sending to", kargs[0])
parent_conn.send({'func': func, 'args': kargs})
taskN += 1
for proc, parent_conn, child_conn in tasks:
retval = parent_conn.recv()
print("ParRegion ret: ", retval)
# assert retval ==
print("Starting local copy in parallel region")
func(*kargs) # Perform the work on this process
print("barrier at end of perallel region")
barrier()
inParallelContext = False
def parForEach(start, # First iteration's index
end, # Final iteration's index
loopBody, # Function to execute each iteration
scheduleType = 'guided', # Load balancing technique
minChunk = 1 # Smallest block of iterations
):
"""Execute the local iterations of a decomposed loop with
the specified scheduling.
"""
global myID, libems, EMSmmapID, _regionN, pinThreads, domainName, inParallelContext, tasks, nThreads
if scheduleType == 'static': # Evenly divide the iterations across threads
blocksz = math.floor((end - start) / nThreads) + 1
s = int((blocksz * myID) + start)
e = int((blocksz * (myID + 1)) + start)
if e > end:
e = end
for idx in range(s, e):
loopBody(idx)
else: # scheduleType == 'dynamic' or scheduleType == 'guided' or default
# Initialize loop bounds, block size, etc.
if scheduleType == 'guided':
ems_sched_type = 1200 # From ems.h
elif scheduleType == 'dynamic':
ems_sched_type = 1201 # From ems.h
else:
print("EMS parForEach: Invalid scheduling type:", scheduleType)
return
libems.EMSloopInit(EMSmmapID, start, end, minChunk, ems_sched_type)
# Do not enter loop until all threads have completed initialization
# If the barrier is present at the loop end, this may replaced
# with first-thread initialization.
barrier()
extents = _loop_chunk()
while extents['end'] - extents['start'] > 0:
for idx in range(extents['start'], extents['end']):
loopBody(idx)
extents = _loop_chunk()
# Do not proceed until all iterations have completed
# TODO: OpenMP offers NOWAIT to skip this barrier
barrier()
def tmStart(emsElems):
"""Start a Transaction
Input is an array containing EMS elements to transition from
Full to Empty:
arr = [ [ emsArr0, idx0 ], [ emsArr1, idx1, true ], [ emsArr2, idx2 ] ]
"""
global myID, libems, EMSmmapID, _regionN, pinThreads, domainName, inParallelContext, tasks, nThreads
def getElemUID(item):
# print("item=", item[0], item[1], item[0].index2key(item[1]) )
# return (item[0].mmapID * MAX_ELEM_PER_REGION) + item[0].index2key(item[1])
return str(item[0].mmapID * MAX_ELEM_PER_REGION) + str(item[1])
MAX_ELEM_PER_REGION = 10000000000000
# Build a list of indexes of the elements to lock which will hold
# the elements sorted for deadlock free acquisition
# Sort the elements to lock according to a global ordering
sortedElems = sorted(emsElems, key=getElemUID)
# Acquire the locks in the deadlock free order, saving the contents
# of the memory when it locked.
# Mark read-write data as Empty, and read-only data under a readers-writer lock
tmHandle = []
for elem in sortedElems:
if len(elem) > 2:
val = elem[0].readRW(elem[1])
readonly = True
else:
val = elem[0].readFE(elem[1])
readonly = False
tmHandle.append([elem[0], elem[1], readonly, val])
return tmHandle
def tmEnd(
tmHandle, # The returned value from tmStart
doCommit # Commit or Abort the transaction
):
"""Commit or abort a transaction
The tmHandle contains the result from tmStart:
[ [ EMSarray, index, isReadOnly, origValue ], ... ]
"""
global myID, libems, EMSmmapID, _regionN, pinThreads, domainName, inParallelContext, tasks, nThreads
for emsElem in tmHandle:
if doCommit:
if emsElem[2]:
# Is a read-only element
emsElem[0].releaseRW(emsElem[1])
else:
# Is read-write, keep current value and mark Full
emsElem[0].setTag(emsElem[1], 'full')
else:
# Abort the transaction
if emsElem[2]:
# Is read-only element
emsElem[0].releaseRW(emsElem[1])
else:
# Write back the original value and mark full
emsElem[0].writeEF(emsElem[1], emsElem[3])
# -------------------------------------------------
def critical(func, timeout=1000000):
"""Serialize execution through this function"""
global myID, libems, EMSmmapID, _regionN, pinThreads, domainName, inParallelContext, tasks, nThreads
libems.EMScriticalEnter(EMSmmapID, timeout)
retObj = func()
libems.EMScriticalExit(EMSmmapID)
return retObj
def master(func):
"""Perform func only on thread 0"""
global myID, libems, EMSmmapID, _regionN, pinThreads, domainName, inParallelContext, tasks, nThreads
if myID == 0:
return func()
def single(func):
"""Perform the function func once by the first thread to reach
the function. The final barrier is required because a
thread may try to execute the next single-execution region
before other threads have finished this region, which the EMS
runtime cannot tell apart. Barriers are phased, so a barrier
is used to prevent any thread from entering the next single-
execution region before this one is complete
"""
global myID, libems, EMSmmapID, _regionN, pinThreads, domainName, inParallelContext, tasks, nThreads
retObj = None
if libems.EMSsingleTask(EMSmmapID):
retObj = func()
barrier()
return retObj
def barrier(timeout=10000):
"""Wrapper around the EMS global barrier"""
global myID, libems, EMSmmapID, _regionN, pinThreads, domainName, inParallelContext, tasks, nThreads
if inParallelContext:
return libems.EMSbarrier(EMSmmapID, timeout)
else:
return 1 # Return a positive non-zero integer -- as if time left on clock
def new(arg0=None, # Maximum number of elements the EMS region can hold
heapSize=0, # #bytes of memory reserved for strings/arrays/objs/maps/etc
filename=None): # Optional filename for persistent EMS memory
"""Creating a new EMS memory region"""
global myID, libems, EMSmmapID, _regionN, pinThreads, domainName, inParallelContext, tasks, nThreads
fillIsJSON = False
emsDescriptor = EMSarray( # Internal EMS descriptor
nElements=1, # Required: Maximum number of elements in array
heapSize=0, # Optional, default=0: Space, in bytes, for strings, maps, objects, etc.
mlock=0, # Optional, 0-100% of EMS memory into RAM
useMap=False, # Optional, default=False: Use a map from keys to indexes
useExisting=False, # Optional, default=false: Preserve data if a file already exists
persist=True, # Optional, default=true: Preserve the file after threads exit
doDataFill=False, # Optional, default=false: Data values should be initialized
dataFill=None, # Optional, default=false: Value to initialize data to
dimStride=[] # Stride factors for each dimension of multidimensional arrays
)
emsDescriptor.dataFill = _new_EMSval(None)
if arg0 is None: # Nothing passed in, assume length 1
emsDescriptor.dimensions = [1]
else:
if type(arg0) == dict: # User passed in emsArrayDescriptor
if 'dimensions' in arg0:
if type(arg0['dimensions']) == list:
emsDescriptor.dimensions = arg0['dimensions']
else:
emsDescriptor.dimensions = [arg0['dimensions']]
if 'heapSize' in arg0:
emsDescriptor.heapSize = arg0['heapSize']
if 'mlock' in arg0:
emsDescriptor.mlock = arg0['mlock']
if 'useMap' in arg0:
emsDescriptor.useMap = arg0['useMap']
if 'filename' in arg0:
emsDescriptor.filename = arg0['filename']
if 'persist' in arg0:
emsDescriptor.persist = arg0['persist']
if 'useExisting' in arg0:
emsDescriptor.useExisting = arg0['useExisting']
if 'doDataFill' in arg0:
if arg0['doDataFill']:
emsDescriptor.doDataFill = True
emsDescriptor.dataFill = _new_EMSval(arg0['dataFill'])
if 'doSetFEtags' in arg0:
emsDescriptor.doSetFEtags = arg0['doSetFEtags']
if 'setFEtags' in arg0:
if (arg0['setFEtags'] == 'full'):
emsDescriptor.setFEtagsFull = True
else:
emsDescriptor.setFEtagsFull = False
else:
if type(arg0) == list: # User passed in multi-dimensional array
emsDescriptor.dimensions = arg0
else:
if type(arg0) == int: # User passed in scalar 1-D array length
emsDescriptor.dimensions = [arg0]
else:
print("EMSnew: ERROR Non-integer type of arg0", str(arg0), type(arg0))
if heapSize > 0:
emsDescriptor.heapSize = heapSize
if emsDescriptor.heapSize <= 0 and emsDescriptor.useMap:
print("WARNING: New EMS array with no heap, disabling mapped keys")
emsDescriptor.useMap = False
if type(filename) == str:
emsDescriptor.filename = filename
if not emsDescriptor.dimensions:
emsDescriptor.dimensions = [emsDescriptor.nElements]
# Compute the stride factors for each dimension of a multidimensal array
for dim in emsDescriptor.dimensions:
emsDescriptor.dimStride.append(emsDescriptor.nElements)
emsDescriptor.nElements *= dim
# Name the region if a name wasn't given
if not emsDescriptor.filename:
emsDescriptor.filename = '/EMS_region_' + str(_regionN)
emsDescriptor.persist = False
if emsDescriptor.useExisting:
try:
fh = open(str(emsDescriptor.filename), 'r')
fh.close()
except Exception as err:
print("EMS ERROR: file " + str(emsDescriptor.filename) + " should already exist, but does not. " + str(err))
return
# init() is first called from thread 0 to perform one-thread
# only operations (ie: unlinking an old file, opening a new
# file). After thread 0 has completed initialization, other
# threads can safely share the EMS array.
if not emsDescriptor.useExisting and myID != 0:
barrier()
emsDescriptor.mmapID = libems.EMSinitialize(
emsDescriptor.nElements,
emsDescriptor.heapSize, # 1
emsDescriptor.useMap,
emsDescriptor.filename.encode(), # 3
emsDescriptor.persist,
emsDescriptor.useExisting, # 5
emsDescriptor.doDataFill,
fillIsJSON,
emsDescriptor.dataFill,
emsDescriptor.doSetFEtags,
emsDescriptor.setFEtagsFull,
myID, pinThreads, nThreads,
emsDescriptor.mlock
)
if not emsDescriptor.useExisting and myID == 0:
barrier()
_regionN += 1
barrier() # Wait until all processes finished initialization
return emsDescriptor
'''
# ================================================
def ems_thread_stub(taskN, conn):
global myID, libems, EMSmmapID, _regionN, pinThreads, domainName, inParallelContext, tasks, nThreads
myID = taskN
while True:
print("THread Stub waiting for message", taskN, conn)
mesg = conn.recv()
try:
print("THread Stub message:", taskN, *mesg)
mesg['func'](*mesg['args'])
except:
print("Had exception in the stub", taskN)
'''
# =======================================================================================
def _new_EMSval(val):
global myID, libems, EMSmmapID, _regionN, pinThreads, domainName, inParallelContext, tasks, nThreads
global global_weakkeydict
emsval = ffi.new("EMSvalueType *")
emsval[0].length = 0
if type(val) == str:
if sys.version_info[0] == 2: # Python 2 or 3
newval = ffi.new('char []', bytes(val))
else:
newval = ffi.new('char []', bytes(val, 'utf-8'))
emsval[0].value = newval
emsval[0].length = len(newval) + 1
emsval[0].type = TYPE_STRING
global_weakkeydict[emsval] = (emsval[0].length, emsval[0].type, emsval[0].value, newval)
elif type(val) == int:
emsval[0].type = TYPE_INTEGER
emsval[0].value = ffi.cast('void *', val)
elif type(val) == float:
emsval[0].type = TYPE_FLOAT
ud_tmp = ffi.new('EMSulong_double *')
ud_tmp[0].d = ffi.cast('double', val)
emsval[0].value = ffi.cast('void *', ud_tmp[0].u64)
elif type(val) == bool:
emsval[0].type = TYPE_BOOLEAN
emsval[0].value = ffi.cast('void *', val)
elif type(val) == list:
if sys.version_info[0] == 2: # Python 2 or 3
newval = ffi.new('char []', bytes(json.dumps(val)))
else:
newval = ffi.new('char []', bytes(json.dumps(val), 'utf-8'))
emsval[0].value = newval
emsval[0].length = len(newval) + 1
emsval[0].type = TYPE_JSON
global_weakkeydict[emsval] = (emsval[0].length, emsval[0].type, emsval[0].value, newval)
elif type(val) == dict:
if sys.version_info[0] == 2: # Python 2 or 3
newval = ffi.new('char []', bytes(json.dumps(val)))
else:
newval = ffi.new('char []', bytes(json.dumps(val), 'utf-8'))
emsval[0].value = newval
emsval[0].length = len(newval) + 1
emsval[0].type = TYPE_JSON
global_weakkeydict[emsval] = (emsval[0].length, emsval[0].type, emsval[0].value, newval)
elif val is None:
emsval[0].type = TYPE_UNDEFINED
emsval[0].value = ffi.cast('void *', 0xdeadbeef)
else:
print("EMS ERROR - unknown type of value:", type(val), val)
return None
return emsval
# ==========================================================================================
class EMSarray(object):
def __init__(self,
nElements=1, # Required: Maximum number of elements in array
heapSize=0, # Optional, default=0: Space, in bytes, for strings, maps, objects, etc.
mlock=0, # Optional, 0-100% of EMS memory into RAM
useMap=False, # Optional, default=false: Use a map from keys to indexes
useExisting=False, # Optional, default=false: Preserve data if a file already exists
persist=True, # Optional, default=true: Preserve the file after threads exit
doDataFill=False, # Optional, default=false: Data values should be initialized
dataFill=None, # Optional, default=false: Value to initialize data to
dimStride=[] # Stride factors for each dimension of multidimensional arrays
):
global myID, libems, EMSmmapID, _regionN, pinThreads, domainName, inParallelContext, tasks, nThreads
# self.ems = ems
self.nElements = nElements
self.heapSize = heapSize
self.mlock = mlock
self.useMap = useMap
self.useExisting = useExisting
self.persist = persist
self.doSetFEtags = True
self.setFEtags = 'full'
self.doDataFill = doDataFill
self.dataFill = dataFill
self.dimStride = dimStride
self.dimensions = None
self.filename = None
self.mmapID = -1
self.mlock = 1
self.doSetFEtags = False # Optional, initialize full/empty tags
self.setFEtagsFull = True # Optional, used only if doSetFEtags is true
# set any attributes here - before initialisation
# these remain as normal attributes
# self.attribute = attribute
# dict.__init__(self, {})
self.__initialised = True
# after initialisation, setting attributes is the same as setting an item
def destroy(self, unlink_file):
"""Release all resources associated with an EMS memory region"""
global myID, libems, EMSmmapID, _regionN, pinThreads, domainName, inParallelContext, tasks, nThreads
barrier()
if myID == 0:
libems.EMSdestroy(self.mmapID, unlink_file)
barrier()
def _returnData(self, emsval):
global myID, libems, EMSmmapID, _regionN, pinThreads, domainName, inParallelContext, tasks, nThreads
if emsval[0].type == TYPE_STRING:
if sys.version_info[0] == 2: # Python 2 or 3
return ffi.string(ffi.cast("char *", emsval[0].value))
else:
return ffi.string(ffi.cast('char *', emsval[0].value)).decode('utf-8')
elif emsval[0].type == TYPE_JSON:
if sys.version_info[0] == 2: # Python 2 or 3
json_str = ffi.string(ffi.cast("char *", emsval[0].value))
else:
json_str = ffi.string(ffi.cast('char *', emsval[0].value)).decode('utf-8')
tmp_str = "{\"data\":" + json_str + "}"
return json.loads(tmp_str)['data']
elif emsval[0].type == TYPE_INTEGER:
return int(ffi.cast('int64_t', emsval[0].value))
elif emsval[0].type == TYPE_FLOAT:
ud_tmp = ffi.new('EMSulong_double *')
ud_tmp[0].u64 = ffi.cast('uint64_t', emsval[0].value)
return ud_tmp[0].d
elif emsval[0].type == TYPE_BOOLEAN:
return bool(int(ffi.cast('uint64_t', emsval[0].value)))
elif emsval[0].type == TYPE_UNDEFINED:
return None
else:
print("EMS ERROR - unknown type of value:", type(emsval), emsval)
return None
def sync(self):
"""Synchronize memory with storage"""
global myID, libems, EMSmmapID, _regionN, pinThreads, domainName, inParallelContext, tasks, nThreads
return libems.EMSsync(self.mmapID)
def index2key(self, index):
global myID, libems, EMSmmapID, _regionN, pinThreads, domainName, inParallelContext, tasks, nThreads
key = _new_EMSval(None)
assert libems.EMSindex2key(self.mmapID, index, key)
return self._returnData(key)
# ==================================================================
# Wrappers around Stacks and Queues
def push(self, value):
nativeValue = _new_EMSval(value)
return libems.EMSpush(self.mmapID, nativeValue)
def pop(self):
val = _new_EMSval(None)
libems.EMSpop(self.mmapID, val)
return self._returnData(val)
def dequeue(self):
val = _new_EMSval(None)
libems.EMSdequeue(self.mmapID, val)
return self._returnData(val)
def enqueue(self, value):
nativeValue = _new_EMSval(value)
return libems.EMSenqueue(self.mmapID, nativeValue)
# ==================================================================
# Wrappers around Primitive AMOs
# Translate EMS maps and multi-dimensional array indexes/keys
# into EMS linear addresses
# Apparently it is illegal to pass a native function as an argument
def write(self, indexes, value):
nativeIndex = _new_EMSval(self._idx(indexes))
nativeValue = _new_EMSval(value)
libems.EMSwrite(self.mmapID, nativeIndex, nativeValue)
return (self.mmapID, nativeIndex, nativeValue)
def writeEF(self, indexes, value):
nativeIndex = _new_EMSval(self._idx(indexes))
nativeValue = _new_EMSval(value)
libems.EMSwriteEF(self.mmapID, nativeIndex, nativeValue)
return (self.mmapID, nativeIndex, nativeValue)
def writeXF(self, indexes, value):
nativeIndex = _new_EMSval(self._idx(indexes))
nativeValue = _new_EMSval(value)
libems.EMSwriteXF(self.mmapID, nativeIndex, nativeValue)
return (self.mmapID, nativeIndex, nativeValue)
def writeXE(self, indexes, value):
nativeIndex = _new_EMSval(self._idx(indexes))
nativeValue = _new_EMSval(value)
libems.EMSwriteXE(self.mmapID, nativeIndex, nativeValue)
return (self.mmapID, nativeIndex, nativeValue)
# ---------------------------------------------------
def read(self, indexes):
emsnativeidx = _new_EMSval(self._idx(indexes))
val = _new_EMSval(None)
libems.EMSread(self.mmapID, emsnativeidx, val)
return self._returnData(val)
def readFE(self, indexes):
emsnativeidx = _new_EMSval(self._idx(indexes))
val = _new_EMSval(None)
libems.EMSreadFE(self.mmapID, emsnativeidx, val)
return self._returnData(val)
def readFF(self, indexes):
emsnativeidx = _new_EMSval(self._idx(indexes))
val = _new_EMSval(None)
libems.EMSreadFF(self.mmapID, emsnativeidx, val)
return self._returnData(val)
def readRW(self, indexes):
emsnativeidx = _new_EMSval(self._idx(indexes))
val = _new_EMSval(None)
libems.EMSreadRW(self.mmapID, emsnativeidx, val)
return self._returnData(val)
def releaseRW(self, indexes):
emsnativeidx = _new_EMSval(self._idx(indexes))
return libems.EMSreleaseRW(self.mmapID, emsnativeidx)
def setTag(self, indexes, fe):
emsnativeidx = _new_EMSval(self._idx(indexes))
return libems.EMSsetTag(self.mmapID, emsnativeidx, (fe == 'full'))
# ---------------------------------------------------
# Atomic RMW
def faa(self, indexes, val):
if type(val) == dict:
print("EMSfaa ERROR: Cannot add an object to something")
return val
else:
ems_nativeidx = _new_EMSval(self._idx(indexes))
ems_val = _new_EMSval(val)
ems_retval = _new_EMSval(None)
assert libems.EMSfaa(self.mmapID, ems_nativeidx, ems_val, ems_retval)
return self._returnData(ems_retval)
def cas(self, indexes, oldVal, newVal):
if type(oldVal) == dict:
print("EMScas ERROR: Cannot compare objects, only JSON primitives")
return None
else:
ems_nativeidx = _new_EMSval(self._idx(indexes))
ems_oldval = _new_EMSval(oldVal)
ems_newval = _new_EMSval(newVal)
ems_retval = _new_EMSval(None)
libems.EMScas(self.mmapID, ems_nativeidx, ems_oldval, ems_newval, ems_retval)
return self._returnData(ems_retval)
def _idx(self, indexes):
idx = 0
if type(indexes) == list: # Is a Multidimension array: [x,y,z]
rank = 0
for index in indexes:
idx += index * self.dimStride[rank]
rank += 1
else:
if not type(indexes) == int and not self.useMap: # If no map, only use integers
print("EMS ERROR: Non-integer index used, but EMS memory was not configured to use a map (useMap)",
indexes, type(indexes), self.useMap)
idx = -1
else: # Is a mappable intrinsic type
idx = indexes
return idx
def __getattr__(self, attr):
return self.read(attr)
def __setattr__(self, attr, value):
if not '_EMSarray__initialised' in self.__dict__ or attr in self.__dict__:
# Ignore object initialization and normal attribute access
return dict.__setattr__(self, attr, value)
else:
return self.write(attr, value)
def __getitem__(self, item):
return EMSelement(self, item)
def __setitem__(self, key, value):
return self.write(key, value)
# =============================================================================================
class EMSelement(object):
def __init__(self, ems_array, index):
self._ems_array = ems_array
self._index = index
def write(self, value):
return self._ems_array.write(self._index, value)
def writeXE(self, value):
return self._ems_array.writeXE(self._index, value)
def writeXF(self, value):
return self._ems_array.writeXF(self._index, value)
def writeEF(self, value):
return self._ems_array.writeEF(self._index, value)
def read(self):
return self._ems_array.read(self._index)
def readFF(self):
return self._ems_array.readFF(self._index)
def readFE(self):
return self._ems_array.readFE(self._index)
def releaseRW(self):
return self._ems_array.releaseRW(self._index)
def readRW(self):
return self._ems_array.readRW(self._index)
def setTag(self, fe):
return self._ems_array.setTag(self._index, fe)
def faa(self, value):
return self._ems_array.faa(self._index, value)
def cas(self, oldVal, newVal):
return self._ems_array.cas(self._index, oldVal, newVal)
================================================
FILE: Python/setup.py
================================================
"""
+-----------------------------------------------------------------------------+
| Extended Memory Semantics (EMS) Version 1.6.1 |
| Synthetic Semantics http://www.synsem.com/ mogill@synsem.com |
+-----------------------------------------------------------------------------+
| Copyright (c) 2016, Jace A Mogill. All rights reserved. |
| |
| Redistribution and use in source and binary forms, with or without |
| modification, are permitted provided that the following conditions are met: |
| * Redistributions of source code must retain the above copyright |
| notice, this list of conditions and the following disclaimer. |
| * Redistributions in binary form must reproduce the above copyright |
| notice, this list of conditions and the following disclaimer in the |
| documentation and/or other materials provided with the distribution. |
| * Neither the name of the Synthetic Semantics nor the names of its |
| contributors may be used to endorse or promote products derived |
| from this software without specific prior written permission. |
| |
| THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS |
| "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT |
| LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR |
| A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL SYNTHETIC |
| SEMANTICS LLC BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, |
| EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, |
| PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR |
| PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF |
| LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING |
| NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS |
| SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. |
| |
+-----------------------------------------------------------------------------+
"""
from distutils.core import setup, Extension
import os
import sys
# Path to C library source code
src_path = '../src/'
# OS Specific link flags
link_args = []
if sys.platform == "linux" or sys.platform == "linux2":
link_args.append("-lrt")
elif sys.platform == "darwin":
pass
else:
pass
setup(
name="ems",
version="1.6.1",
py_modules=["ems"],
setup_requires=["cffi>=1.0.0"],
install_requires=["cffi>=1.0.0"],
# Author details
author='Jace A Mogill',
author_email='jace@mogill.com',
description='Extended Memory Semantics (EMS) for Python',
license='BSD',
# The project's main homepage.
url='https://github.com/mogill/ems',
ext_modules=[Extension('libems.so',
[src_path + filename for filename in
['collectives.cc', 'ems.cc', 'ems_alloc.cc', 'loops.cc', 'primitives.cc', 'rmw.cc']],
extra_link_args=link_args
)],
long_description='Persistent Shared Memory and Parallel Programming Model',
keywords=["non volatile memory",
"NVM",
"NVMe",
"multithreading",
"multithreaded",
"parallel",
"parallelism",
"concurrency",
"shared memory",
"multicore",
"manycore",
"transactional memory",
"TM",
"persistent memory",
"pmem",
"Extended Memory Semantics",
"EMS"]
)
================================================
FILE: README.md
================================================
OSX | Linux | Node 4.1-14.x, Python2/3:
[](https://travis-ci.org/mogill/ems)
[](https://www.npmjs.com/package/ems)
### [API Documentation](http://mogill.github.io/ems/Docs/reference.html) | [EMS Website](http://mogill.github.io/ems/Docs/index.html)
# Extended Memory Semantics (EMS)
__EMS makes possible persistent shared memory parallelism between Node.js, Python, and C/C++__.
Extended Memory Semantics (EMS) unifies synchronization and storage primitives
to address several challenges of parallel programming:
+ Allows any number or kind of processes to share objects
+ Manages synchronization and object coherency
+ Implements persistence to non-volatile memory and secondary storage
+ Provides dynamic load-balancing between processes
+ May substitute or complement other forms of parallelism
## [Examples: Parallel web servers, word counting](https://github.com/mogill/ems/tree/master/Examples)
#### Table of Contents
* [Parallel Execution Models Supported](#Types-of-Concurrency) Fork Join, Bulk Synchronous Parallel, User defined
* [Atomic Operations](#Built-in-Atomic-Operations) Atomic Read-Modify-Write operations
* [Examples](https://github.com/mogill/ems/tree/master/Examples) Parallel web servers, word counting
* [Benchmarks](#Examples-and-Benchmarks) Bandwidth, Transaction processing
* [Synchronization as a Property of the Data, Not a Duty for Tasks](#Synchronization-Property) Full/Empty tags
* [Installation](#Installation) Downloading from Git or NPM
* [Roadmap](#Roadmap) The Future™! It's all already happened
#### EMS is targeted at tasks too large for one core or one process but too small for a scalable cluster
A modern multi-core server has 16-32 cores and nearly 1TB of memory,
equivalent to an entire rack of systems from a few years ago.
As a consequence, jobs formerly requiring a Map-Reduce cluster
can now be performed entirely in shared memory on a single server
without using distributed programming.
## Sharing Persistent Objects Between Python and Javascript
Inter-language example in [interlanguage.{js,py}](https://github.com/mogill/ems/tree/master/Examples/Interlanguage)
The animated GIF demonstrates the following steps:
* Start Node.js REPL, create an EMS memory
* Store "Hello"
* Open a second session, begin the Python REPL
* Connect Python to the EMS shared memory
* Show the object created by JS is present in Python
* Modify the object, and show the modification can be seen in JS
* Exit both REPLs so no programs are running to "own" the EMS memory
* Restart Python, show the memory is still present
* Initialize a counter from Python
* Demonstrate atomic Fetch and Add in JS
* Start a loop in Python incrementing the counter
* Simultaneously print and modify the value from JS
* Try to read "empty" data from Python, the process blocks
* Write the empty memory, marking it full, Python resumes execution
## Types of Concurrency
EMS extends application capabilities to include transactional memory and
other fine-grained synchronization capabilities.
EMS implements several different parallel execution models:
Fork-Join Multiprocess: execution begins with a single process that creates new processes
when needed, those processes then wait for each other to complete.
Bulk Synchronous Parallel: execution begins with each process starting the program at the
main entry point and executing all the statements
User Defined: parallelism may include ad-hoc processes and mixed-language applications
## Built in Atomic Operations
EMS operations may performed using any JSON data type, read-modify-write operations
may use any combination of JSON data types.
like operations on ordinary data.
Atomic read-modify-write operations are available
in all concurrency modes, however collectives are not
available in user defined modes.
- __Atomic Operations__:
Read, write, readers-writer lock, read when full and atomically mark empty, write when empty and atomically mark full
- __Primitives__:
Stacks, queues, transactions
- __Read-Modify-Write__:
Fetch-and-Add, Compare and Swap
- __Collective Operations__:
All basic [OpenMP](https://en.wikipedia.org/wiki/OpenMP)
collective operations are implemented in EMS:
dynamic, block, guided, as are the full complement of static loop scheduling,
barriers, master and single execution regions
## Examples and Benchmarks
### [API Documentation](http://mogill.github.io/ems/Docs/reference.html) | [EMS Website](http://mogill.github.io/ems/Docs/index.html)
### Word Counting Using Atomic Operations
[Word counting example](https://github.com/mogill/ems/tree/master/Examples)
Map-Reduce is often demonstrated using word counting because each document can
be processed in parallel, and the results of each document's dictionary reduced
into a single dictionary. This EMS implementation also
iterates over documents in parallel, but it maintains a single shared dictionary
across processes, atomically incrementing the count of each word found.
The final word counts are sorted and the most frequently appearing words
are printed with their counts.
The performance of this program was measured using an Amazon EC2 instance:
`c4.8xlarge (132 ECUs, 36 vCPUs, 2.9 GHz, Intel Xeon E5-2666v3, 60 GiB memory`
The leveling of scaling around 16 cores despite the presence of ample work
may be related to the use of non-dedicated hardware:
Half of the 36 vCPUs are presumably HyperThreads or otherwise shared resource.
AWS instances are also bandwidth limited to EBS storage, where our Gutenberg
corpus is stored.
### Bandwidth Benchmarking
[STREAMS Example](https://github.com/mogill/ems/tree/master/Examples/STREAMS)
A benchmark similar to [STREAMS](https://www.cs.virginia.edu/stream/)
gives us the maximum speed EMS double precision
floating point operations can be performed on a
`c4.8xlarge (132 ECUs, 36 vCPUs, 2.9 GHz, Intel Xeon E5-2666v3, 60 GiB memory`.
### Benchmarking of Transactions and Work Queues
[Transactions and Work Queues Example](https://github.com/mogill/ems/tree/master/Examples)
Transactional performance is measured alone, and again with a separate
process appending new processes as work is removed from the queue.
The experiments were run using an Amazon EC2 instance: c4.8xlarge (132 ECUs, 36 vCPUs, 2.9 GHz, Intel Xeon E5-2666v3, 60 GiB memory
#### Experiment Design
Six EMS arrays are created, each holding 1,000,000 numbers. During the
benchmark, 1,000,000 transactions are performed, each transaction involves 1-5
randomly selected elements of randomly selected EMS arrays.
The transaction reads all the elements and
performs a read-modify-write operation involving at least 80% of the elements.
After all the transactions are complete, the array elements are checked
to confirm all the operations have occurred.
The parallel process scheduling model used is *block dynamic* (the default),
where each process is responsible for successively smaller blocks
of iterations. The execution model is *bulk synchronous parallel*, each
processes enters the program at the same main entry point
and executes all the statements in the program.
`forEach` loops have their normal semantics of performing all iterations,
`parForEach` loops are distributed across threads, each process executing
only a portion of the total iteration space.
Immediate Transactions: Each process generates a transaction on integer data then immediately performs it.
Transactions from a Queue: One of the processes generates the individual transactions and appends
them to a work queue the other threads get work from.
Note: As the number of processes increases, the process generating the transactions
and appending them to the work queue is starved out by processes performing transactions,
naturally maximizing the data access rate.
Immediate Transactions on Strings: Each process generates a transaction appending to
a string, and then immediately performs the transaction.
Measurements
Elem. Ref'd: Total number of elements read and/or written
Table Updates: Number of different EMS arrays (tables) written to
Trans. Performed: Number of transactions performed across all EMS arrays (tables)
Trans. Enqueued: Rate transactions are added to the work queue (only 1 generator thread in these experiments)
## [Synchronization as a Property of the Data, Not a Duty for Tasks](#Synchronization-Property)
### [API Documentation](http://mogill.github.io/ems/Docs/reference.html) | [EMS Website](http://mogill.github.io/ems/Docs/index.html)
EMS internally stores tags that are used for synchronization of
user data, allowing synchronization to happen independently of
the number or kind of processes accessing the data. The tags
can be thought of as being in one of three states, _Empty,
Full,_ or _Read-Only_, and the EMS intrinsic functions
enforce atomic access through automatic state transitions.
The EMS array may be indexed directly using an integer, or using a key-index
mapping from any primitive type. When a map is used, the key and data
itself are updated atomically.
EMS memory is an array of JSON values
(Number, Boolean, String, Undefined, or Object) accessed using atomic
operators and/or transactional memory. Safe parallel access
is managed by passing through multiple gates: First mapping a
key to an index, then accessing user data protected by EMS
tags, and completing the whole operation atomically.
EMS Data Tag Transitions & Atomic operations:
F=Full, E=Empty, X=Don't Care, RW=Readers-Writer lock (# of current readers)
CAS=Compare-and-Swap, FAA=Fetch-and-Add
## More Technical Information
For a more complete description of the principles of operation,
contact the author at ems@rotang.com
[ Complete API reference ](https://github.com/mogill/ems/tree/master/Docs/reference.html)
## Installation
Because all systems are already multicore,
parallel programs require no additional equipment, system permissions,
or application services, making it easy to get started.
The reduced complexity of
lightweight threads communicating through shared memory
is reflected in a rapid code-debug cycle for ad-hoc application development.
### Quick Start with the Makefile
To build and test all C, Python 2 and 3, and Node.js targets,
a makefile can automate most build and test tasks.
```sh
dunlin> make help
Extended Memory Semantics -- Build Targets
===========================================================
all Build all targets, run all tests
node Build only Node.js
py Build both Python 2 and 3
py[2|3] Build only Python2 or 3
test Run both Node.js and Py tests
test[_js|_py|_py2|_py3] Run only Node.js, or only Py tests, respectively
clean Remove all files that can be regenerated
clean[_js|_py|_py2|_py3] Remove Node.js or Py files that can be regenerated
```
### Install via npm
EMS is available as a NPM Package. EMS depends on the Node addon API
(N-API) package.
```sh
npm install ems
```
### Install via GitHub
Download the source code, then compile the native code:
```sh
git clone https://github.com/mogill/ems.git
cd ems
npm install
```
### Installing for Python
Python users should download and install EMS git (see above).
There is no PIP package, but not due lack of desire or effort.
A pull request is most welcome!
### Run Some Examples
Click here for __[Detailed Examples](https://github.com/mogill/ems/tree/master/Examples)__.
On a Mac and most Linux
distributions EMS will "just work", but
some Linux distributions restrict access to shared memory. The
quick workaround is to run jobs as root, a long-term solution will
vary with Linux distribution.
Run the work queue driven transaction processing example on 8 processes:
```sh
npm run
```
Or manually via:
```sh
cd Examples
node concurrent_Q_and_TM.js 8
```
Running all the tests with 8 processes:
```sh
npm run test # Alternatively: npm test
```
```sh
cd Tests
rm -f EMSthreadStub.js # Do not run the machine generated script used by EMS
for test in `ls *js`; do node $test 8; done
```
## Platforms Supported
As of 2016-05-01, Mac/Darwin and Linux are supported. A windows port pull request is welcomed!
## Roadmap
EMS 1.0 uses Nan for long-term Node.js support, we continue to
develop on OSX and Linux via Vagrant.
EMS 1.3 introduces a C API.
EMS 1.4 Python API
EMS 1.4.8 Improved examples and documentation
EMS 1.5 Refactored JS-EMS object conversion temporary storage
EMS 1.6 **[This Release]** Updated to replace deprecated NodeJS NAN API with the N-API.
EMS 1.7 **[Planned]** Key deletion that frees all resources. Replace open hashing with chaining.
EMS 1.8 **[Planned]** Memory allocator based on
*R. Marotta, M. Ianni, A. Scarselli, A. Pellegrini and F. Quaglia, "NBBS: A Non-Blocking Buddy System for Multi-core Machines," 2019 19th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGRID), Larnaca, Cyprus, 2019, pp. 11-20, doi: 10.1109/CCGRID.2019.00011.*
EMS 1.9 **[Planned]** Vectorized JSON indexer.
EMS 1.10 **[Planned]** Support for [persistent main system memory (PMEM)](http://pmem.io/) when
multiple processes are supported.
EMS 2.0 **[Planned]** New API which more tightly integrates with
ES6, Python, and other dynamically typed languages languages,
making atomic operations on persistent memory more transparent.
## License
BSD, other commercial and open source licenses are available.
## Links
[API Documentation](http://mogill.github.io/ems/Docs/reference.html)
[EMS Website](http://mogill.github.io/ems/Docs/index.html)
[Download the NPM](https://www.npmjs.org/package/ems)
[Get the source at GitHub](https://github.com/mogill/ems)
## About
Jace A Mogill specializes in hardware/software co-design of
resource constrained computing at both the largest and smallest scales.
He has over 20 years experience with distributed, multi-core, FPGA, CGRA, GPU, CPU,
and custom computer architectures.
###### Copyright (C)2011-2020 Jace A Mogill
================================================
FILE: Tests/3dSpace.js
================================================
/*-----------------------------------------------------------------------------+
| Extended Memory Semantics (EMS) Version 1.4.0 |
| Synthetic Semantics http://www.synsem.com/ mogill@synsem.com |
+-----------------------------------------------------------------------------+
| Copyright (c) 2011-2014, Synthetic Semantics LLC. All rights reserved. |
| Copyright (c) 2015-2016, Jace A Mogill. All rights reserved. |
| |
| Redistribution and use in source and binary forms, with or without |
| modification, are permitted provided that the following conditions are met: |
| * Redistributions of source code must retain the above copyright |
| notice, this list of conditions and the following disclaimer. |
| * Redistributions in binary form must reproduce the above copyright |
| notice, this list of conditions and the following disclaimer in the |
| documentation and/or other materials provided with the distribution. |
| * Neither the name of the Synthetic Semantics nor the names of its |
| contributors may be used to endorse or promote products derived |
| from this software without specific prior written permission. |
| |
| THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS |
| "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT |
| LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR |
| A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL SYNTHETIC |
| SEMANTICS LLC BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, |
| EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, |
| PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR |
| PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF |
| LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING |
| NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS |
| SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. |
| |
+-----------------------------------------------------------------------------*/
var ems = require('ems')(parseInt(process.argv[2]), false);
var assert = require('assert');
var start;
var dim1 = ems.new(1000);
var dims3d = [800, 120, 50];
var dims2d = [5000, 500];
var dim3 = ems.new(dims3d, 0, "/tmp/EMS_3d_space");
var dim2 = ems.new(dims2d, 800000000);
var idx = ems.myID;
var i, j, k;
dim1.write(idx * 10, idx * 100);
var v = dim1.read(idx * 10);
assert(v == ems.myID * 100, "Since 1d write-read failed");
function val3d(i, j, k) {
return i + (j * 10000) + (k * 100000);
}
//-------------------------------------------------------------------
// Timer function
function stopTimer(timer, nOps, label) {
function fmtNumber(n) {
var s = ' ' + n.toString().replace(/\B(?=(\d{3})+(?!\d))/g, ",");
if (n < 1) return n;
else {
return s.substr(s.length - 15, s.length);
}
}
ems.master(function () {
var now = new Date().getTime();
var x = (nOps * 1000000) / ((now - timer) * 1000);
ems.diag(fmtNumber(nOps) + label + fmtNumber(Math.floor(x).toString()) + " ops/sec");
})
}
//------------------------------------------------------------------------------
// Spinup loop to physically allocate all memory as RW
ems.barrier();
start = new Date().getTime();
for (k = ems.myID; k < dims3d[2]; k += ems.nThreads) {
for (j = 0; j < dims3d[1]; j += 1) {
for (i = 0; i < dims3d[0]; i += 1) {
dim3.write([i, j, k], val3d(i, j, k));
}
}
}
ems.barrier();
stopTimer(start, dims3d[0] * dims3d[1] * dims3d[2], " First touch ");
//------------------------------------------------------------------------------
// Test read & write on all nodes
ems.barrier();
start = new Date().getTime();
for (k = ems.myID; k < dims3d[2]; k += ems.nThreads) {
for (j = 0; j < dims3d[1]; j += 1) {
for (i = 0; i < dims3d[0]; i += 1) {
dim3.write([i, j, k], val3d(k, j, i)); // different pattern;
}
}
}
for (k = ems.myID; k < dims3d[2]; k += ems.nThreads) {
for (j = 0; j < dims3d[1]; j += 1) {
for (i = 0; i < dims3d[0]; i += 1) {
assert(dim3.read([i, j, k]) === val3d(k, j, i),
"Failed to verify parallel 3D data " + dim3.read([i, j, k]) +
" != " + val3d(k, j, i) + " " + i + " " + j + " " + k);
}
}
}
ems.barrier();
stopTimer(start, 2 * dims3d[0] * dims3d[1] * dims3d[2], " Read/Write ops ");
//------------------------------------------------------------------------------
// How long would it take node 0 alone on native array?
ems.barrier();
start = new Date().getTime();
ems.master(function () {
var native = new Array(dims3d[0] * dims3d[1] * dims3d[2]);
for (k = 0; k < dims3d[2]; k += 1) {
for (j = 0; j < dims3d[1]; j += 1) {
for (i = 0; i < dims3d[0]; i += 1) {
var idx = 0;
dims3d.forEach(function (x, i) {
idx += x * dims3d[i]
});
native[idx] = val3d(i, j, k)
}
}
}
});
ems.barrier();
stopTimer(start, dims3d[0] * dims3d[1] * dims3d[2], " native array ");
//------------------------------------------------------------------------------
// Critical Regions
dim1.write(30, 3333333);
var prev = dim1.read(30);
ems.barrier();
start = new Date().getTime();
var nIters = Math.floor(1000000 / ems.nThreads);
for (i = 0; i < nIters; i++) {
ems.critical(function () {
var x = dim1.read(30);
x++;
dim1.write(30, x);
}, 1000); // TODO: Write proper fail case test for critical timeout
}
ems.barrier();
stopTimer(start, nIters * ems.nThreads, " critical regions ");
ems.master(function () {
assert(dim1.read(30) === (prev + (ems.nThreads * nIters)),
"Critical region was racing x=" + dim1.read(30) + " sum=" + (prev + (ems.nThreads * nIters)) +
" prev=" + prev);
});
//------------------------------------------------------------------------------
// Purge D2
start = new Date().getTime();
for (j = ems.myID; j < dims2d[1]; j += ems.nThreads) {
for (i = 0; i < dims2d[0]; i += 1) {
dim2.writeXE([i, j], -val3d(i + 10, j + 10, 0));
}
}
ems.barrier();
stopTimer(start, dims2d[0] * dims2d[1], " writeXF purges ");
//------------------------------------------------------------------------------
// ReadFE then WriteEF
start = new Date().getTime();
if (ems.myID != 0) {
for (j = ems.myID; j < dims2d[1]; j += ems.nThreads) {
for (i = 0; i < dims2d[0]; i += 1) {
assert(dim2.readFF([i, j]) === val3d(i + 10, j + 10, 0),
"Failed to verify 2D FE data: " +
dim2.readFF([i, j]) + " " + val3d(i + 10, j + 10, 0) + " i-j: " + i + " " + j);
}
}
} else {
for (j = 0; j < dims2d[1]; j += 1) {
for (i = 0; i < dims2d[0]; i += 1) {
dim2.writeEF([i, j], val3d(i + 10, j + 10, 0));
}
}
for (i = 0; i < dims2d[0]; i += 1) {
assert(dim2.readFF([i, 0]) === val3d(i + 10, 10, 0),
"Failed to verify 2D FE node 0 data: " +
dim2.readFF([i, 0]) + " " + val3d(i + 10, 10, 0) + " i-j: " + i + " " + 0);
}
}
ems.barrier();
stopTimer(start, 2 * dims2d[0] * dims2d[1], " FE-EF Dataflow ");
//---------------------------------------------------------------
// Redo dataflow but using strings
start = new Date().getTime();
for (j = ems.myID; j < dims2d[1]; j += ems.nThreads) {
for (i = 0; i < dims2d[0]; i += 1) {
dim2.writeXE([i, j], 'mem' + (-1 * val3d(i + 10, j + 10, 0)));
}
}
ems.barrier();
stopTimer(start, dims2d[0] * dims2d[1], " XF srting purge ");
//------------------------------------------------------------------------------
// ReadFE then WriteEF
start = new Date().getTime();
if (ems.myID != 0) {
for (j = ems.myID; j < dims2d[1]; j += ems.nThreads) {
for (i = 0; i < dims2d[0]; i += 1) {
assert(dim2.readFF([i, j]) === 'mem' + (val3d(i + 10, j + 10, 0)),
"Failed to verify 2D string FE data: " +
dim2.readFF([i, j]) + " " + val3d(i + 10, j + 10, 0) + " i-j: " + i + " " + j);
}
}
} else {
for (j = 0; j < dims2d[1]; j += 1) {
for (i = 0; i < dims2d[0]; i += 1) {
dim2.writeEF([i, j], 'mem' + val3d(i + 10, j + 10, 0))
}
}
for (i = 0; i < dims2d[0]; i += 1) {
assert(dim2.readFF([i, 0]) === 'mem' + val3d(i + 10, 10, 0),
"Failed to verify 2D FE node 0 data: " +
dim2.readFF([i, 0]) + " " + val3d(i + 10, 10, 0) + " i-j: " + i + " " + 0);
}
}
ems.barrier();
stopTimer(start, 2 * dims2d[0] * dims2d[1], " Dataflow w/strgs ");
================================================
FILE: Tests/CASdataFlow.js
================================================
/*-----------------------------------------------------------------------------+
| Extended Memory Semantics (EMS) Version 1.0.0 |
| Synthetic Semantics http://www.synsem.com/ mogill@synsem.com |
+-----------------------------------------------------------------------------+
| Copyright (c) 2011-2014, Synthetic Semantics LLC. All rights reserved. |
| Copyright (c) 2015-2016, Jace A Mogill. All rights reserved. |
| |
| Redistribution and use in source and binary forms, with or without |
| modification, are permitted provided that the following conditions are met: |
| * Redistributions of source code must retain the above copyright |
| notice, this list of conditions and the following disclaimer. |
| * Redistributions in binary form must reproduce the above copyright |
| notice, this list of conditions and the following disclaimer in the |
| documentation and/or other materials provided with the distribution. |
| * Neither the name of the Synthetic Semantics nor the names of its |
| contributors may be used to endorse or promote products derived |
| from this software without specific prior written permission. |
| |
| THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS |
| "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT |
| LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR |
| A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL SYNTHETIC |
| SEMANTICS LLC BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, |
| EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, |
| PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR |
| PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF |
| LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING |
| NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS |
| SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. |
| |
+-----------------------------------------------------------------------------*/
'use strict';
var ems = require('ems')(parseInt(process.argv[2]));
var assert = require('assert');
var start, i;
//-------------------------------------------------------------------
// Timer function
function stopTimer(timer, nOps, label) {
function fmtNumber(n) {
var s = ' ' + n.toString().replace(/\B(?=(\d{3})+(?!\d))/g, ",");
if (n < 1) {
return n;
}
return s.substr(s.length - 15, s.length);
}
ems.master(function () {
var now = new Date().getTime();
var x = (nOps * 1000000) / ((now - timer) * 1000);
ems.diag(fmtNumber(nOps) + label + fmtNumber(Math.floor(x).toString()) + " ops/sec");
});
}
String.prototype.getBytes = function () {
var bytes = [];
for (i = 0; i < this.length; i += 1) {
bytes.push(this.charCodeAt(i));
}
return bytes;
};
//------------------------------------------------------------------------------
// Compare and Swap numbers
var casBuf = ems.new(ems.nThreads, 100000, '/tmp/EMS_3dstrings');
casBuf.writeXF(ems.myID, 100000 + ems.myID);
var v = casBuf.readFF(ems.myID);
assert(v === 100000 + ems.myID, "Bad first readback v=" + v + " computed=" + 100000 + ems.myID);
v = 2000000 + casBuf.readFE(ems.myID);
casBuf.writeEF(ems.myID, v);
assert(casBuf.readFE(ems.myID) == 2100000 + ems.myID, "Bad second readback");
casBuf.writeEF(ems.myID, 33333);
casBuf.readFE(ems.myID);
casBuf.writeEF(ems.myID, 44444);
casBuf.readFE(ems.myID);
casBuf.writeEF(ems.myID, "six");
casBuf.readFF(ems.myID);
assert(casBuf.readFF(ems.myID) === "six", "Six wasn't the same... Len mem/literal" + casBuf.readFF(ems.myID).length + "/" + "six".length );
var third = 'third' + ems.myID;
casBuf.readFE(ems.myID);
casBuf.writeEF(ems.myID, third);
v = casBuf.readFF(ems.myID);
assert(v === third,
"Bad third (string) readback v=|" +
v + "| bytes:" + v.getBytes() + " Type=" + typeof v + " length="+ v.length + " |||| computed=|" +
third + "| bytes:" + third.getBytes() + " type=" + typeof third + " length="+ third.length);
v = casBuf.cas(ems.myID, third, 'fourth' + ems.myID);
assert(casBuf.readFE(ems.myID) === 'fourth' + ems.myID,
"Bad fourth (string) mem=" + v + "| old=" + third + "| computed=" + 'fourth' + ems.myID);
casBuf.writeEF(ems.myID, 100 * ems.myID);
assert(casBuf.readFF(ems.myID) === 100 * ems.myID, "Failed to go from strings to numbers");
ems.barrier();
casBuf.writeXF(0, 0);
ems.barrier();
start = new Date().getTime();
var nIters = 50;
for (i = 0; i < nIters; i += 1) {
var oldVal = -123;
while (oldVal != ems.myID) {
oldVal = casBuf.cas(0, ems.myID, (ems.myID + 1) % ems.nThreads);
}
}
stopTimer(start, nIters * ems.nThreads, " CAS Numbers ");
ems.barrier();
//------------------------------------------------------------------------------
// Compare and Swap strings
ems.master(function () {
casBuf.writeXF(0, 'test0');
});
ems.barrier();
start = new Date().getTime();
nIters = 5;
var nAMOs = 0;
for (i = 0; i < nIters; i += 1) {
oldVal = "no match";
var testMe = 'test' + ems.myID;
var testNext = 'test' + ((ems.myID + 1) % ems.nThreads);
while (oldVal != testMe) {
oldVal = casBuf.cas(0, testMe, testNext);
nAMOs++;
}
}
ems.barrier();
var rff = casBuf.readFF(0);
assert(rff === 'test0', "Incorrect final CAS string: " + rff + " test0");
ems.diag("Performed " + nAMOs + " AMOs");
stopTimer(start, nIters * ems.nThreads, " CAS Strings ");
ems.barrier();
//------------------------------------------------------------------------------
// Clobbering old casBuf definition forces destructor to be called
casBuf = ems.new(1, 10000, '/tmp/EMS_3dstrings'); // TODO : memory allocator for strings and objects
================================================
FILE: Tests/ES6/harmony_proxies.js
================================================
/*-----------------------------------------------------------------------------+
| Extended Memory Semantics (EMS) Version 1.4.0 |
| Synthetic Semantics http://www.synsem.com/ mogill@synsem.com |
+-----------------------------------------------------------------------------+
| Copyright (c) 2016, Jace A Mogill. All rights reserved. |
| |
| Redistribution and use in source and binary forms, with or without |
| modification, are permitted provided that the following conditions are met: |
| * Redistributions of source code must retain the above copyright |
| notice, this list of conditions and the following disclaimer. |
| * Redistributions in binary form must reproduce the above copyright |
| notice, this list of conditions and the following disclaimer in the |
| documentation and/or other materials provided with the distribution. |
| * Neither the name of the Synthetic Semantics nor the names of its |
| contributors may be used to endorse or promote products derived |
| from this software without specific prior written permission. |
| |
| THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS |
| "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT |
| LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR |
| A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL SYNTHETIC |
| SEMANTICS LLC BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, |
| EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, |
| PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR |
| PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF |
| LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING |
| NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS |
| SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. |
| |
+-----------------------------------------------------------------------------*/
"use strict";
var ems = require("ems")(1);
var assert = require("assert");
// Create a temporary EMS space to demonstrate read/write operations
var emsData = ems.new({
dimensions: 1000,
heapSize: 20000,
useMap: true,
ES6proxies: true,
useExisting: false
});
emsData["foo"] = 123;
assert(emsData["foo"] === 123);
assert(emsData.foo === 123);
emsData.bar = 321;
assert(emsData["bar"] === 321);
assert(emsData.bar === 321);
assert(emsData.readFE("bar") === 321);
assert(emsData.bar === 321);
emsData.writeEF("bar", "last write");
assert(emsData.readFE("bar") === "last write");
assert(emsData.bar === "last write");
console.log("All operations passed.");
================================================
FILE: Tests/accum_omp.c
================================================
/*-----------------------------------------------------------------------------+
| Extended Memory Semantics (EMS) Version 1.0.0 |
| Synthetic Semantics http://www.synsem.com/ mogill@synsem.com |
+-----------------------------------------------------------------------------+
| Copyright (c) 2011-2014, Synthetic Semantics LLC. All rights reserved. |
| Copyright (c) 2015-2016, Jace A Mogill. All rights reserved. |
| |
| Redistribution and use in source and binary forms, with or without |
| modification, are permitted provided that the following conditions are met: |
| * Redistributions of source code must retain the above copyright |
| notice, this list of conditions and the following disclaimer. |
| * Redistributions in binary form must reproduce the above copyright |
| notice, this list of conditions and the following disclaimer in the |
| documentation and/or other materials provided with the distribution. |
| * Neither the name of the Synthetic Semantics nor the names of its |
| contributors may be used to endorse or promote products derived |
| from this software without specific prior written permission. |
| |
| THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS |
| "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT |
| LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR |
| A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL SYNTHETIC |
| SEMANTICS LLC BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, |
| EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, |
| PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR |
| PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF |
| LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING |
| NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS |
| SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. |
| |
+-----------------------------------------------------------------------------*/
#include
#include
#define SIZE 3200
int main() {
int i;
int a[ SIZE ];
for(i = 0; i < SIZE; i++) a[i] = 0;
#define NITER 100000000
#pragma omp parallel for
for(i = 0; i < NITER; i++) {
int retval = __sync_fetch_and_add( &(a[i% SIZE]), 1 );
}
int sum = 0;
for(i = 0; i < SIZE; i++) {
sum += a[i];
}
printf("sum = %d\n", sum);
return(0);
}
================================================
FILE: Tests/barrier.js
================================================
/*-----------------------------------------------------------------------------+
| Extended Memory Semantics (EMS) Version 1.4.0 |
| Synthetic Semantics http://www.synsem.com/ mogill@synsem.com |
+-----------------------------------------------------------------------------+
| Copyright (c) 2011-2014, Synthetic Semantics LLC. All rights reserved. |
| Copyright (c) 2015-2016, Jace A Mogill. All rights reserved. |
| |
| Redistribution and use in source and binary forms, with or without |
| modification, are permitted provided that the following conditions are met: |
| * Redistributions of source code must retain the above copyright |
| notice, this list of conditions and the following disclaimer. |
| * Redistributions in binary form must reproduce the above copyright |
| notice, this list of conditions and the following disclaimer in the |
| documentation and/or other materials provided with the distribution. |
| * Neither the name of the Synthetic Semantics nor the names of its |
| contributors may be used to endorse or promote products derived |
| from this software without specific prior written permission. |
| |
| THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS |
| "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT |
| LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR |
| A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL SYNTHETIC |
| SEMANTICS LLC BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, |
| EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, |
| PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR |
| PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF |
| LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING |
| NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS |
| SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. |
| |
+-----------------------------------------------------------------------------*/
'use strict';
var ems = require('ems')(process.argv[2]);
var assert = require('assert');
var iter, idx, memVal;
var nIters = Math.floor(100000 / process.argv[2]);
var sums = ems.new(ems.nThreads);
sums.writeXF(ems.myID, 0);
for (iter = 0; iter < nIters; iter += 1) {
ems.barrier();
idx = (ems.myID + iter) % ems.nThreads;
memVal = sums.read(idx);
assert(memVal === iter,
"myID=" + ems.myID + " iter=" + iter + " memval=" + memVal);
sums.write(idx, memVal + 1);
}
ems.barrier(100);
for (iter = 0; iter < nIters; iter += 1) {
ems.barrier();
idx = (ems.myID + iter) % ems.nThreads;
memVal = sums.read(idx);
sums.write(idx, memVal + 1);
}
ems.barrier();
assert(sums.read(ems.myID) === (nIters * 2),
"myID=" + ems.myID + " wrong final memval=" + sums.read(ems.myID) + " expected=" + (nIters * 2));
ems.barrier();
var shared = ems.new(process.argv[2]*2, 0, "/tmp/EMS_mynewFoo");
shared.write(0, 0);
ems.barrier();
var tmp = shared.read(0);
assert(tmp === 0, "Didn't initialize to 0, got " + tmp);
ems.barrier();
shared.faa(0, 1);
ems.barrier();
assert(shared.read(0) === ems.nThreads, "Didn't count (" + shared.read(0) + ") to nnodes (" + ems.nThreads + ")");
ems.barrier();
if (ems.myID === 0) {
shared.write(0, 0);
}
ems.barrier();
assert(shared.read(0) === 0, "reinit Didn't initialize to 0, got " + shared.read(0));
ems.barrier();
var nIter = 10000;
var i;
for (i = 0; i < nIter; i += 1) {
shared.faa(0, 1);
}
ems.barrier();
assert(shared.read(0) === ems.nThreads * nIter, "Random wait FAA failed: count (" + shared.read(0) + ") to nnodes (" + nIter + ")");
ems.barrier();
if (ems.myID === 0) {
shared.write(0, 0);
}
ems.barrier();
if (ems.myID !== 0) {
assert(shared.read(0) === 0, "Didn't pass sync after clearing");
}
ems.barrier();
// nIter = 10000 / process.argv[2];
for (i = 0; i < nIter; i += 1) {
shared.faa(0, 1);
}
ems.barrier();
var sr = shared.read(0);
ems.barrier();
assert(sr === ems.nThreads * nIter, "Fast Looped FAA failed: count (" + sr + ") to nnodes (" + nIter + ")");
var timeout = -1;
try {
timeout = ems.barrier(0);
console.log("Barrier timeout succeeded but should have failed");
} catch (err) {
ems.diag("Correctly timed out at barrier:", err);
}
assert(timeout <= 0);
================================================
FILE: Tests/check_image_files.sh
================================================
#!/bin/bash
# -*- coding: utf-8, tab-width: 2 -*-
function check_image_files () {
local SELFPATH="$(readlink -m "$BASH_SOURCE"/..)"
cd -- "$SELFPATH"/.. || return $?
local IMG_DIFF="$(diff -sU 0 <(find_images_used) <(find_images_available
) | sed -re '1d;2d;/^@@/d' | norm_sort_paths)"
local IMG_UNUSED=() IMG_MISS=()
readarray -t IMG_UNUSED < <(<<<"$IMG_DIFF" sed -nre 's!^\+!!p')
echo "unused images: ${#IMG_UNUSED[@]}"
printf ' * %s\n' "${IMG_UNUSED[@]}"
readarray -t IMG_MISS < <(<<<"$IMG_DIFF" sed -nre 's!^\-!!p')
local N_MISS="${#IMG_MISS[@]}"
[ "$N_MISS" == 0 ] || exec >&2
echo "missing images: $N_MISS"
printf ' ! %s\n' "${IMG_MISS[@]}"
[ "$N_MISS" == 0 ] || return 4
}
function norm_sort_paths () {
sed -re '
s~(^|/)\./~\1~g
' | sort --version-sort --unique
}
function find_images_available () {
local FIND_OPT=(
.
-type f
'(' -false
-o -name '*.png'
-o -name '*.svg'
-o -name '*.jpeg'
')'
)
find "${FIND_OPT[@]}" | norm_sort_paths
}
function find_images_used () {
local HTML_FILES=()
readarray -t HTML_FILES < <(find . -name '*.html')
grep -HoPe ']*>' -- "${HTML_FILES[@]}" | sed -re '
s~\s+~ ~g
s~= end) {
console.log("Index out of bounds?!");
}
for (var i = 0; i < nIters; i++) {
sum++;
}
}
stats.writeXF(0, 0);
ems.barrier();
startTime = util.timerStart();
ems.parForEach(start, end, doWork);
stats.faa(0, sum);
util.timerStop(startTime, n * ems.nThreads, " Default ", ems.myID);
ems.barrier();
ems.diag("The sum: " + sum);
ems.barrier();
sum = stats.read(0);
assert(sum == n, "Default scheduling failed sum=" + sum + " expected " + n);
sum = 0;
startTime = util.timerStart();
ems.parForEach(start, end, doWork, 'dynamic');
util.timerStop(startTime, n * ems.nThreads, " Dynamic ", ems.myID);
ems.barrier();
ems.diag("The sum: " + sum);
ems.barrier();
sum = stats.read(0);
assert(sum == n, "Dynamic scheduling failed sum=" + sum + " expected " + n);
sum = 0;
startTime = util.timerStart();
ems.parForEach(start, end, doWork, 'static');
util.timerStop(startTime, n * ems.nThreads, " Static ", ems.myID);
ems.barrier();
ems.diag("The sum: " + sum);
ems.barrier();
sum = stats.read(0);
assert(sum == n, "Static scheduling failed sum=" + sum + " expected " + n);
sum = 0;
startTime = util.timerStart();
ems.parForEach(start, end, doWork, 'guided', 20);
util.timerStop(startTime, n * ems.nThreads, " Guided by 20 ", ems.myID);
ems.barrier();
ems.diag("The sum: " + sum);
ems.barrier();
sum = stats.read(0);
assert(sum == n, "Guided scheduling failed sum=" + sum + " expected " + n);
================================================
FILE: Tests/mapped_test.js
================================================
/*-----------------------------------------------------------------------------+
| Extended Memory Semantics (EMS) Version 1.3.0 |
| Synthetic Semantics http://www.synsem.com/ mogill@synsem.com |
+-----------------------------------------------------------------------------+
| Copyright (c) 2011-2014, Synthetic Semantics LLC. All rights reserved. |
| Copyright (c) 2015-2016, Jace A Mogill. All rights reserved. |
| |
| Redistribution and use in source and binary forms, with or without |
| modification, are permitted provided that the following conditions are met: |
| * Redistributions of source code must retain the above copyright |
| notice, this list of conditions and the following disclaimer. |
| * Redistributions in binary form must reproduce the above copyright |
| notice, this list of conditions and the following disclaimer in the |
| documentation and/or other materials provided with the distribution. |
| * Neither the name of the Synthetic Semantics nor the names of its |
| contributors may be used to endorse or promote products derived |
| from this software without specific prior written permission. |
| |
| THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS |
| "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT |
| LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR |
| A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL SYNTHETIC |
| SEMANTICS LLC BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, |
| EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, |
| PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR |
| PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF |
| LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING |
| NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS |
| SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. |
| |
+-----------------------------------------------------------------------------*/
'use strict';
var ems = require('ems')(parseInt(process.argv[2]), false);
var assert = require('assert');
var arrayElem = ['abcd', 1234.567, true, 987, -1234.567, -987, -986, -988];
var arrLen = arrayElem.length;
var objMap = ems.new({
dimensions: [arrLen],
heapSize: arrLen * 200,
useMap: true,
useExisting: false,
setFEtags: 'full'
});
arrayElem.forEach(function (elem, idx) {
if (ems.myID === 0) { console.log("Trying to map:", elem, typeof elem); }
objMap.writeXF(elem, elem + idx);
var readback = objMap.readFF(elem);
assert(readback === elem + idx, 'Readback of ' + (elem + idx) + 'didnt match:' + readback);
});
================================================
FILE: Tests/py_api.py
================================================
# -*- coding: utf-8 -*-
"""
+-----------------------------------------------------------------------------+
| Extended Memory Semantics (EMS) Version 1.4.1 |
| Synthetic Semantics http://www.synsem.com/ mogill@synsem.com |
+-----------------------------------------------------------------------------+
| Copyright (c) 2016, Jace A Mogill. All rights reserved. |
| |
| Redistribution and use in source and binary forms, with or without |
| modification, are permitted provided that the following conditions are met: |
| * Redistributions of source code must retain the above copyright |
| notice, this list of conditions and the following disclaimer. |
| * Redistributions in binary form must reproduce the above copyright |
| notice, this list of conditions and the following disclaimer in the |
| documentation and/or other materials provided with the distribution. |
| * Neither the name of the Synthetic Semantics nor the names of its |
| contributors may be used to endorse or promote products derived |
| from this software without specific prior written permission. |
| |
| THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS |
| "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT |
| LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR |
| A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL SYNTHETIC |
| SEMANTICS LLC BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, |
| EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, |
| PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR |
| PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF |
| LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING |
| NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS |
| SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. |
| |
+-----------------------------------------------------------------------------+
"""
import sys
import os
import time
sys.path.append("../Python/ems/") # Path to EMS Python Module
import ems
nprocs = 12
nelem = 1000
ems.initialize(nprocs, False, 'bsp', '/tmp/pyems_foo_test.ems')
time.sleep(ems.myID/5)
assert ems.barrier()
various_consts = ["簣ひょ갤妣ゃキェぱ覣.𐤦ぴ盥", -99999, 1 << 35, 9821.3456, "bar", False]
unmapped_fname = '/tmp/py_unmapped.ems'
# TODO: 3D array
unmapped = ems.new(nelem * nprocs, nprocs * nelem * 100, unmapped_fname)
ems.diag("=============== INIT COMPLETE ======================")
def sum(left, right):
if type(left) == int:
return right + left
if type(left) == float:
return right * left
else:
return str(right) + str(left)
def target(idx):
return idx + (ems.myID * nelem)
for const in various_consts:
for idx in range(nelem):
unmapped.writeXF(target(idx), sum(const, idx))
for idx in range(nelem):
val = unmapped.readFE(target(idx))
if sum(const, idx) != val:
ems.diag("I'm lost... sum=%s val=%s idx=%d target=%d" % (sum(const, idx), val, idx, target(idx)))
assert sum(const, idx) == val
ems.barrier()
ems.diag("Starting mapped tests")
arrLen = 1000
mapped_fname = '/tmp/py_mapped.ems'
mapped = ems.new({
'dimensions': [arrLen],
'heapSize': arrLen * 200,
'useMap': True,
'useExisting': False,
'filename': mapped_fname,
'doSetFEtags': True,
'setFEtags': 'empty',
'dataFill': None
})
arrayElem = ['abcd', True, 1234.567, False, {'x': 'xxx', 'y': 'yyyyy'}, 987, None, [10, 11, 12, 13]]
objElem = {'a': 1, 'b': 321.653, 'x': {'nesta': False, 'nestb': None}, 'c': 'asdasd'}
teststr = 'magic!!string?!'
mapped.writeEF(teststr, ems.myID)
assert ems.myID == mapped.readFE(teststr)
# ems.diag("First mapped write passed")
mapped.writeEF(teststr + "XXX", ems.myID * 100)
assert ems.myID * 100 == mapped.readFE(teststr + "XXX")
# ems.diag("Second mapped write passed")
mapped.writeEF(ems.myID, teststr)
# ems.diag("Wrote the test string")
assert teststr == mapped.readFE(ems.myID)
# Check read/writing JSON
mapped.writeEF(ems.myID, objElem)
tmp = mapped.readFE(ems.myID)
# ems.diag("ObjElem READback" + str(tmp))
assert objElem == tmp
mapped.writeEF(ems.myID, arrayElem)
tmp = mapped.readFE(ems.myID)
# ems.diag("READback:" + str(mapped.readFE(ems.myID)))
assert arrayElem == tmp
tmp = mapped.read(ems.myID)
assert arrayElem == tmp
mapped.write(ems.myID, 'dummy')
assert 'dummy' == mapped.read(ems.myID)
# ==========================================================================
def parfun(idx):
i = mapped.faa('index sum', idx - start + 1)
c = mapped.faa('counter', 1)
timeout = 10000
while (timeout > 0) and mapped.cas('cas test', None, 'something') is not None:
timeout -= 1
assert timeout > 0
assert 'something' == mapped.cas('cas test', 'something', None)
# ems.diag("PARFUN: idx(%d) i(%d) c(%d)" % (idx, i, c))
def triangular(max = 10):
for top in range(1, max):
x = 0
for idx in range(1, top+1):
x += idx
print("top=", top, " x=", x, " calc=", (top * (top + 1)) / 2)
assert x == (top * (top + 1)) / 2
def check_tri(msg):
index_sum = mapped.readFF('index sum')
count = mapped.readFF('counter')
# ems.diag(msg + ": sum=%d expected=%d" % (index_sum, (niters * (niters + 1))/2))
assert index_sum == trisum
assert count == niters
reset_tri()
def reset_tri():
ems.barrier()
mapped.writeXF('index sum', 0)
mapped.writeXF('counter', 0)
mapped.writeXF('cas test', None)
ems.barrier()
unmapped.writeXF(ems.myID, 1000+ems.myID)
ems.single(triangular)
idx = (ems.myID+1) % nprocs
assert unmapped.readFF(idx) == 1000+idx
# Scheduling needs proper testing other than visual inspection
start = 100
end = 120
niters = end - start
trisum = (niters * (niters + 1))/2
reset_tri()
ems.parForEach(start, end, parfun)
check_tri("default")
ems.parForEach(start, end, parfun, 'static')
check_tri("static")
ems.parForEach(start, end, parfun, 'guided')
check_tri("guided")
ems.parForEach(start, end, parfun, 'dynamic')
check_tri("dynamic")
# ==========================================================================
unmapped.writeXF(0, True)
unmapped.writeXF(1, 0)
mapped.writeXF('one', 0)
ems.barrier()
tm_data = [[unmapped, 1], [mapped, 'one'], [unmapped, 0, True]]
for idx in range(niters):
transaction = ems.tmStart(tm_data)
unmapped_tmp = unmapped.read(1)
mapped_tmp = mapped.read('one')
assert unmapped.read(0)
unmapped.write(1, unmapped_tmp + 1)
mapped.write('one', mapped_tmp + 1)
ems.tmEnd(transaction, True)
ems.barrier()
assert unmapped.readFF(1) == nprocs * niters
assert mapped.readFF('one') == nprocs * niters
ems.barrier()
# ==========================================================================
def plus_one():
tmp = unmapped.read(0)
unmapped.write(0, tmp + 1)
ems.barrier()
unmapped.writeXF(0, 0)
ems.barrier()
ems.critical(plus_one)
readback = unmapped.readFF(0)
# ems.diag("Initial Readback before critical: %d" % readback)
if readback == nprocs:
unmapped.writeXF(0, -1234)
ems.barrier()
readback = unmapped.readFF(0)
# ems.diag("Readback of critical: %d" % readback)
assert readback == -1234
# ==========================================================================
def check_master():
assert ems.myID == 0
ems.master(check_master)
# ==========================================================================
def check_single():
global idx
oldval = mapped.cas(idx, None, 'Wheee!!!')
if oldval is None:
assert mapped.faa('count', 1) == 0
idx = 987.654
ems.barrier()
mapped.writeXF(idx, None)
mapped.writeXF('count', 0)
ems.barrier()
ems.single(check_single)
assert mapped.readFF(idx) == 'Wheee!!!'
assert mapped.readFF('count') == 1
# ==========================================================================
keys = [mapped.index2key(idx) for idx in range(arrLen)]
# print("KEYS!", keys)
assert 'count' in keys
assert 'counter' in keys
assert teststr in keys
assert teststr+'XXX' in keys
# ==========================================================================
stack = ems.new({
'dimensions': [arrLen],
'heapSize': arrLen * 100,
'useExisting': False,
'filename': '/tmp/stack_test.ems',
'doSetFEtags': True,
'setFEtags': 'empty',
'dataFill': None
})
for elem in arrayElem:
assert stack.push(elem) >= 0
for elem in arrayElem:
assert stack.pop() in arrayElem
ems.barrier()
assert stack.pop() is None
ems.barrier()
for elem in arrayElem:
assert stack.enqueue(elem) >= 0
for elem in arrayElem:
assert stack.dequeue() in arrayElem
ems.barrier()
assert stack.dequeue() is None
ems.barrier()
# ==========================================================================
# Fancy array syntax
mapped.writeXF(-1234, 'zero')
mapped[-1234].writeXF(0)
idx = 'fancy' + str(ems.myID)
val = 123 + ems.myID
rw_val = 12345 + ems.myID
rw_idx = 'rw test' + str(ems.myID)
mapped[rw_idx].writeXF(rw_val)
mapped[idx].writeXF(val)
# print("Fancyread", ems.myID, mapped[idx].read(), mapped[idx].xread)
assert val == mapped[idx].read()
assert val == mapped[idx].readFF()
assert val == mapped[idx].readFE()
val *= 200
mapped[idx].writeEF(val)
assert val == mapped[idx].readFF()
ems.barrier()
assert rw_val == mapped[rw_idx].readRW()
assert mapped[rw_idx].releaseRW() >= 0
ems.barrier()
val *= 200
mapped[idx].write(val)
assert val == mapped[idx].read()
assert 0 <= mapped[-1234].faa(1) < nprocs
if ems.myID == 0:
mapped[-1234].cas(nprocs-1, True)
assert mapped[-1234].read()
mapped[idx].writeXE(123)
mapped[idx].setTag('full')
if ems.myID == 0:
mapped.writeXF('syntax_test', 123)
assert mapped.readFE('syntax_test') == 123
assert mapped['syntax_test'].read() == 123
assert mapped.syntax_test == 123
mapped['syntax_test2'] = 234
mapped['syntax_test2'].setTag('full')
# assert mapped.readFE('syntax_test2') == 234
mapped['syntax_test3'].writeXF(234)
assert mapped.readFE('syntax_test2') == 234
ems.barrier()
'''
assert mapped.fancy0 == 123
assert unmapped[0] == -123
ems.barrier()
if ems.myID == 0:
mapped.fizz = 'asd'
ems.barrier()
assert mapped.fizz == 'asd'
'''
# ==========================================================================
unmapped.destroy(True)
# ems.diag("UNMAPPING:" + unmapped_fname + " Exist?" + os.path.exists(unmapped_fname))
assert not os.path.exists(unmapped_fname)
mapped.destroy(False)
assert os.path.exists(mapped_fname)
mapped = ems.new({
'dimensions': [arrLen],
'heapSize': arrLen * 200,
'useMap': True,
'useExisting': True,
'filename': mapped_fname,
'doSetFEtags': False,
'doDataFill': False
})
assert mapped.readFF(idx) == 123
================================================
FILE: Tests/readers-writer.js
================================================
/*-----------------------------------------------------------------------------+
| Extended Memory Semantics (EMS) Version 1.3.0 |
| Synthetic Semantics http://www.synsem.com/ mogill@synsem.com |
+-----------------------------------------------------------------------------+
| Copyright (c) 2011-2014, Synthetic Semantics LLC. All rights reserved. |
| Copyright (c) 2015-2016, Jace A Mogill. All rights reserved. |
| |
| Redistribution and use in source and binary forms, with or without |
| modification, are permitted provided that the following conditions are met: |
| * Redistributions of source code must retain the above copyright |
| notice, this list of conditions and the following disclaimer. |
| * Redistributions in binary form must reproduce the above copyright |
| notice, this list of conditions and the following disclaimer in the |
| documentation and/or other materials provided with the distribution. |
| * Neither the name of the Synthetic Semantics nor the names of its |
| contributors may be used to endorse or promote products derived |
| from this software without specific prior written permission. |
| |
| THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS |
| "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT |
| LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR |
| A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL SYNTHETIC |
| SEMANTICS LLC BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, |
| EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, |
| PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR |
| PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF |
| LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING |
| NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS |
| SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. |
| |
+-----------------------------------------------------------------------------*/
var assert = require('assert');
var ems = require('ems')(parseInt(process.argv[2]), false);
var a = ems.new(1, 0, '/tmp/EMS_a');
var arrLen = 10000;
var nTimes;
var nLocks;
var count = ems.new({
dimensions: [10],
heapSize: arrLen * 200,
useMap: true,
useExisting: false,
setFEtags: 'full'
});
a.writeXF(0, 123);
count.writeXF(0, 0);
ems.barrier();
for (nTimes = 0; nTimes < 100000; nTimes += 1) {
a.readRW(0);
nLocks = count.faa(0, 1) + 1;
assert(nLocks <= 7, "Too many locks: " + nLocks);
nLocks = count.faa(0, -1) - 1;
assert(nLocks >= 0, "Too few locks: " + nLocks);
for (var i = 0; i < 100; i++) {
nLocks += Math.sin(i);
}
a.releaseRW(0);
}
ems.barrier();
var objMap = ems.new({
dimensions: [arrLen],
heapSize: arrLen * 200,
useMap: true,
useExisting: false,
setFEtags: 'full'
});
['abcd', 1234.567, true, 987].forEach( function(elem) {
objMap.writeXF(elem, elem);
count.writeXF(elem, 0);
});
ems.barrier();
['abcd', 1234.567, true, 987].forEach( function(elem) {
for (var nTimes = 0; nTimes < 10000; nTimes++) {
var readback = objMap.readRW(elem);
assert(readback === elem,
"Multi Reader read wrong data. Expected(" + elem + "), got(" + readback + ")");
var nLocks = count.faa(elem, 1) + 1;
assert(nLocks <= 7, "Too many locks: " + nLocks);
nLocks = count.faa(elem, -1) - 1;
assert(nLocks >= 0, "Too few locks: " + nLocks);
for (var i = 0; i < nLocks * 100; i += 1) {
nLocks += Math.sin(i);
}
objMap.releaseRW(elem);
}
});
================================================
FILE: Tests/refactoring.js
================================================
/*-----------------------------------------------------------------------------+
| Extended Memory Semantics (EMS) Version 1.4.0 |
| Synthetic Semantics http://www.synsem.com/ mogill@synsem.com |
+-----------------------------------------------------------------------------+
| Copyright (c) 2011-2014, Synthetic Semantics LLC. All rights reserved. |
| Copyright (c) 2015-2016, Jace A Mogill. All rights reserved. |
| |
| Redistribution and use in source and binary forms, with or without |
| modification, are permitted provided that the following conditions are met: |
| * Redistributions of source code must retain the above copyright |
| notice, this list of conditions and the following disclaimer. |
| * Redistributions in binary form must reproduce the above copyright |
| notice, this list of conditions and the following disclaimer in the |
| documentation and/or other materials provided with the distribution. |
| * Neither the name of the Synthetic Semantics nor the names of its |
| contributors may be used to endorse or promote products derived |
| from this software without specific prior written permission. |
| |
| THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS |
| "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT |
| LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR |
| A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL SYNTHETIC |
| SEMANTICS LLC BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, |
| EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, |
| PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR |
| PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF |
| LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING |
| NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS |
| SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. |
| |
+-----------------------------------------------------------------------------*/
'use strict';
var util = require('util');
var assert = require('assert');
var ems = require('ems')(1, 0, 'bsp', '/facto.ems');
var array_len = 10000;
var ems_config = {
dimensions : [array_len],
heapSize : 20000,
useMap : false,
useExisting : false,
filename : '/tmp/refactor.ems'
};
var sums = ems.new(ems_config);
ems_config.useMap = true;
ems_config.filename = '/tmp/anotherplace';
var emptyland = ems.new(ems_config);
ems.barrier();
/*
setTimeout(function () {
// console.log("Pre timeout barrier", ems.myID);
ems.barrier();
console.log("Post timeout barrier", ems.myID);
}, ems.myID * 3000);
*/
var magicval = 0x15432100 + ems.myID;
sums.writeXF(ems.myID, magicval);
var readback = sums.readFF(ems.myID);
assert(readback === magicval, "Expected " + magicval.toString() + ", got " + readback.toString());
sums.readFF(ems.myID);
ems.barrier();
ems.barrier();
sums.writeXE(ems.myID, "Nothing to see");
var idx, teststr;
for (idx = 0; idx < 100000; idx += 1) {
teststr = "some text" + idx;
sums.writeEF(ems.myID, teststr);
readback = sums.readFE(ems.myID);
assert(teststr === readback, "Nope...|" + teststr + "|" + readback + "| len=" + teststr.length + "/" + readback.length);
}
for (idx = 0; idx < 100000; idx += 1) {
teststr = 'foo' + (idx * 3 + idx * 7);
readback = emptyland.read(teststr);
assert(readback === undefined, "Read actual data during undefined map test:" + teststr);
}
var oldval;
for (idx = 0; idx < array_len / 40; idx += 1) {
teststr = 'foo' + (idx * 5) + (idx * array_len);
readback = emptyland.read(teststr);
assert(readback === undefined, "CAS phase: Read actual data during undefined map test:" + teststr);
readback = emptyland.read(teststr);
assert(readback === undefined, "CAS phase: Second Read actual data during undefined map test:" + teststr);
oldval = emptyland.cas(teststr, undefined, teststr);
assert(oldval === undefined, "CAS at teststr(" + teststr + ") wasn't undefined -- value was:" + oldval + "/" + (oldval || 0xf00dd00d).toString(16));
var newmemval = emptyland.readFF(teststr);
assert(teststr === newmemval, "wrong post CAS mem value -- value was:" + newmemval + " expected:" + teststr);
}
// Check index2key
var all_types_len = 10;
var all_types = ems.new({
dimensions : [ all_types_len ],
heapSize : 10000, // Optional, default=0: Space, in bytes, for
useMap : true, // Optional, default=false: Map keys to indexes
filename : '/tmp/refactor.ems' // Optional, default=anonymous:
});
var all_data_types = [false, true, 1234, 987.654321, 'hello'];
all_data_types.forEach(function(type) {
console.log("Trying type", type);
all_types.writeXF(type, type);
});
for(idx = 0; idx < all_types_len; idx += 1) {
console.log("Index:" + idx + " -- key:" + all_types.index2key(idx));
}
================================================
FILE: Tests/stack_and_queue.js
================================================
/*-----------------------------------------------------------------------------+
| Extended Memory Semantics (EMS) Version 1.4.0 |
| Synthetic Semantics http://www.synsem.com/ mogill@synsem.com |
+-----------------------------------------------------------------------------+
| Copyright (c) 2011-2014, Synthetic Semantics LLC. All rights reserved. |
| Copyright (c) 2015-2016, Jace A Mogill. All rights reserved. |
| |
| Redistribution and use in source and binary forms, with or without |
| modification, are permitted provided that the following conditions are met: |
| * Redistributions of source code must retain the above copyright |
| notice, this list of conditions and the following disclaimer. |
| * Redistributions in binary form must reproduce the above copyright |
| notice, this list of conditions and the following disclaimer in the |
| documentation and/or other materials provided with the distribution. |
| * Neither the name of the Synthetic Semantics nor the names of its |
| contributors may be used to endorse or promote products derived |
| from this software without specific prior written permission. |
| |
| THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS |
| "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT |
| LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR |
| A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL SYNTHETIC |
| SEMANTICS LLC BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, |
| EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, |
| PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR |
| PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF |
| LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING |
| NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS |
| SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. |
| |
+-----------------------------------------------------------------------------*/
'use strict';
var ems = require('ems')(parseInt(process.argv[2]));
var assert = require('assert');
var arrLen = 1000000;
var a = ems.new({
dimensions: [arrLen],
heapSize: arrLen * 100,
useExisting: false,
filename: '/tmp/EMS_stack',
doSetFEtags: true,
setFEtags: 'empty',
doSetDataFill: true,
dataFill: undefined
});
var b = ems.new({
dimensions: [arrLen],
heapSize: arrLen * 100,
doSetFEtags: true,
setFEtags: 'empty',
doSetDataFill: true,
dataFill: undefined
});
var timeStart, tmp, i, idx;
//-------------------------------------------------------------------
// Timer function
function stopTimer(timer, nOps, label) {
function fmtNumber(n) {
var s = ' ' + n.toString().replace(/\B(?=(\d{3})+(?!\d))/g, ",");
if (n < 1) return n;
else {
return s.substr(s.length - 15, s.length);
}
}
ems.master(function () {
var now = new Date().getTime();
var x = (nOps * 1000000) / ((now - timer) * 1000);
ems.diag(fmtNumber(nOps) + label + fmtNumber(Math.floor(x).toString()) + " ops/sec");
});
}
var x = a.pop();
assert(x === undefined, "Initial: should have been empty: " + x);
ems.barrier();
a.push(1000 + ems.myID);
ems.barrier();
x = a.pop();
assert(x >= 1000, "Expected something >= 1000 on the stack, found " + x);
ems.barrier();
x = a.pop();
assert(x === undefined, "Expected nothing on the stack, found: " + x);
ems.barrier();
a.push(2000.12345 + ems.myID);
a.push('text ' + ems.myID + 'yeah...');
ems.barrier();
assert(a.pop() !== undefined, "Expected text or ### on the stack, was undefined.")
assert(a.pop() !== undefined, "Expected second text or ### on the stack, was undefined.")
ems.barrier();
assert(a.pop() === undefined, "after two pops stack should have been empty");
ems.barrier();
a.push(3330.12345 + ems.myID);
a.push('more text ' + ems.myID + 'yeah...');
assert(a.pop() !== undefined, "Expected more text or more ### on the stack, was undefined.")
assert(a.pop() !== undefined, "Expected second more text or more ### on the stack, was undefined.")
ems.barrier();
assert(a.pop() === undefined, "repeated two pops should have been empty");
var value = b.dequeue();
ems.barrier();
assert(value === undefined, "Intitial b dequeue should have beeen undefined, was " + value);
ems.barrier();
idx = b.enqueue(4000 + ems.myID);
ems.barrier();
ems.master(function () {
tmp = b.dequeue();
var count = 0;
while (tmp !== undefined) {
tmp = b.dequeue();
count++;
}
assert(count === ems.nThreads, "Didn't find enough queued items");
assert(a.pop() === undefined, "dq1: should still have been empty");
});
ems.barrier();
tmp = b.dequeue();
assert(tmp === undefined, "DQ should be mt: " + tmp);
ems.barrier();
ems.master(function () {
for (i = 0; i < arrLen; i++) {
var str = "sequential" + (i * 100);
b.enqueue(str);
}
for (i = 0; i < arrLen; i++) {
tmp = b.dequeue();
assert(tmp === "sequential" + (i * 100), "Iter " + i + " Should have found ... got " + tmp);
}
assert(a.pop() === undefined, "a pop again should have been empty");
});
ems.barrier();
timeStart = new Date().getTime();
ems.parForEach(0, arrLen, function (idx) {
b.enqueue('foo' + idx);
});
stopTimer(timeStart, arrLen, " enqueued ");
timeStart = new Date().getTime();
ems.parForEach(0, arrLen, function () {
b.dequeue();
});
stopTimer(timeStart, arrLen, " dequeued ");
tmp = b.dequeue();
assert(tmp === undefined, "DQ at end should be mt: " + tmp);
var p = ems.new({
dimensions: [arrLen + 1],
heapSize: arrLen * 50,
doSetFEtags: true,
setFEtags: 'empty',
dataFill: undefined
});
timeStart = new Date().getTime();
ems.parForEach(0, arrLen, function (idx) {
p.push('foo' + idx);
});
stopTimer(timeStart, arrLen, " pushed ");
timeStart = new Date().getTime();
ems.parForEach(0, arrLen, function () {
p.pop();
});
stopTimer(timeStart, arrLen, " popped ");
ems.barrier();
var c = [];
timeStart = new Date().getTime();
ems.master(function () {
for (idx = 0; idx < arrLen; idx++) {
c.push('foo' + idx);
}
});
stopTimer(timeStart, arrLen, " native enqueued ");
timeStart = new Date().getTime();
ems.master(function () {
for (idx = 0; idx < arrLen; idx++) {
c.pop();
}
});
stopTimer(timeStart, arrLen, " native dequeued ");
tmp = b.dequeue();
assert(tmp === undefined, "DQ of B after native end should be mt: " + tmp);
/////////////////////////////////////////
timeStart = new Date().getTime();
ems.parForEach(0, arrLen, function (idx) {
b.enqueue(idx);
});
stopTimer(timeStart, arrLen, " int enqueued ");
timeStart = new Date().getTime();
ems.parForEach(0, arrLen, function () {
b.dequeue();
});
stopTimer(timeStart, arrLen, " int dequeued ");
tmp = b.dequeue();
assert(tmp === undefined, "DQend second time should be mt: " + tmp);
p = ems.new({
dimensions: [arrLen + 1],
heapSize: arrLen * 50,
doSetFEtags: true,
setFEtags: 'empty',
dataFill: undefined
});
timeStart = new Date().getTime();
ems.parForEach(0, arrLen, function (idx) {
p.push(idx);
});
stopTimer(timeStart, arrLen, " int pushed ");
timeStart = new Date().getTime();
ems.parForEach(0, arrLen, function () {
p.pop();
});
stopTimer(timeStart, arrLen, " int popped ");
ems.barrier();
c = [];
timeStart = new Date().getTime();
ems.master(function () {
for (idx = 0; idx < arrLen; idx++) {
c.push(idx);
}
});
stopTimer(timeStart, arrLen, " int native enqueued ");
timeStart = new Date().getTime();
ems.master(function () {
for (idx = 0; idx < arrLen; idx++) {
c.pop();
}
});
stopTimer(timeStart, arrLen, " int native dequeued ");
tmp = b.dequeue();
assert(tmp === undefined, "Final deque should be mt: " + tmp);
================================================
FILE: Tests/strcpy.js
================================================
/*-----------------------------------------------------------------------------+
| Extended Memory Semantics (EMS) Version 1.5.0 |
| Synthetic Semantics http://www.synsem.com/ mogill@synsem.com |
+-----------------------------------------------------------------------------+
| Copyright (c) 2011-2014, Synthetic Semantics LLC. All rights reserved. |
| Copyright (c) 2015-2017, Jace A Mogill. All rights reserved. |
| |
| Redistribution and use in source and binary forms, with or without |
| modification, are permitted provided that the following conditions are met: |
| * Redistributions of source code must retain the above copyright |
| notice, this list of conditions and the following disclaimer. |
| * Redistributions in binary form must reproduce the above copyright |
| notice, this list of conditions and the following disclaimer in the |
| documentation and/or other materials provided with the distribution. |
| * Neither the name of the Synthetic Semantics nor the names of its |
| contributors may be used to endorse or promote products derived |
| from this software without specific prior written permission. |
| |
| THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS |
| "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT |
| LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR |
| A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL SYNTHETIC |
| SEMANTICS LLC BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, |
| EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, |
| PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR |
| PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF |
| LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING |
| NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS |
| SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. |
| |
+-----------------------------------------------------------------------------*/
'use strict';
var ems = require('ems')(parseInt(process.argv[2]));
var assert = require('assert');
var maxlen = 20000000;
var stats = ems.new({
filename: '/tmp/EMS_strlen',
dimensions: [10],
heapSize: (2 * maxlen) + 1 + 10, // 1 = Null, 10 = length of key
useMap: true, // Use a key-index mapping, not integer indexes
setFEtags: 'full', // Initial full/empty state of array elements
doDataFill: true, // Initialize data values
dataFill: 0 // Initial value of new keys
});
if (ems.myID === 0) { stats.writeXE('test_str', 123); }
ems.barrier();
function stringFill(x, n) {
var s = '';
for (;;) {
if (n & 1) s += x;
n >>= 1;
if (n) x += x;
else break;
}
return s;
}
for (var len=2; len < maxlen; len = Math.floor(len * 1.5) ) {
if (ems.myID === 0) { console.log("Len = " + len); }
var str = stringFill('x', len);
stats.writeEF('test_str', str);
var readback = stats.readFE('test_str');
assert(readback === str, 'Mismatched string. Expected len ' + str.length + ' got ' + readback.length);
ems.barrier();
}
for (var len=maxlen; len >= 1; len = Math.floor(len * 0.666) ) {
if (ems.myID === 0) { console.log("Len = " + len); }
var str = stringFill('y', len);
stats.writeEF('test_str', str);
var readback = stats.readFE('test_str');
assert(readback === str, 'Mismatched string. Expected len ' + str.length + ' got ' + readback.length);
ems.barrier();
}
================================================
FILE: Tests/stringTags.js
================================================
/*-----------------------------------------------------------------------------+
| Extended Memory Semantics (EMS) Version 1.0.0 |
| Synthetic Semantics http://www.synsem.com/ mogill@synsem.com |
+-----------------------------------------------------------------------------+
| Copyright (c) 2011-2014, Synthetic Semantics LLC. All rights reserved. |
| Copyright (c) 2015-2016, Jace A Mogill. All rights reserved. |
| |
| Redistribution and use in source and binary forms, with or without |
| modification, are permitted provided that the following conditions are met: |
| * Redistributions of source code must retain the above copyright |
| notice, this list of conditions and the following disclaimer. |
| * Redistributions in binary form must reproduce the above copyright |
| notice, this list of conditions and the following disclaimer in the |
| documentation and/or other materials provided with the distribution. |
| * Neither the name of the Synthetic Semantics nor the names of its |
| contributors may be used to endorse or promote products derived |
| from this software without specific prior written permission. |
| |
| THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS |
| "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT |
| LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR |
| A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL SYNTHETIC |
| SEMANTICS LLC BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, |
| EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, |
| PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR |
| PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF |
| LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING |
| NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS |
| SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. |
| |
+-----------------------------------------------------------------------------*/
var ems = require('ems')(parseInt(process.argv[2]), false);
var assert = require('assert');
var data = ems.new(100, 10000, '/tmp/EMS_sanity');
//data.writeXF(ems.myID, "init"+ems.myID)
data.writeXF(ems.myID, 100 + ems.myID);
var readback = data.readFF(ems.myID);
var computed = "init" + ems.myID;
assert(data.readFF(ems.myID) === 100 + ems.myID, "Readback FF was wrong: " + readback + " != " + computed);
ems.barrier();
assert(data.readFE(ems.myID) === 100 + (ems.myID) % ems.nThreads, "Bad wrap one " + data.read(ems.myID) + " " +
(100 + ((ems.myID) % ems.nThreads)));
data.writeEF((ems.myID + 2) % ems.nThreads, "second" + ems.myID);
readback = data.readFF(ems.myID);
computed = "second" + (ems.myID - 2 + ems.nThreads) % ems.nThreads;
assert(readback === computed, "Bad snake shift: " + readback + " != " + computed);
================================================
FILE: Tests/testUtils.js
================================================
/*-----------------------------------------------------------------------------+
| Extended Memory Semantics (EMS) Version 1.0.0 |
| Synthetic Semantics http://www.synsem.com/ mogill@synsem.com |
+-----------------------------------------------------------------------------+
| Copyright (c) 2011-2014, Synthetic Semantics LLC. All rights reserved. |
| Copyright (c) 2015-2016, Jace A Mogill. All rights reserved. |
| |
| Redistribution and use in source and binary forms, with or without |
| modification, are permitted provided that the following conditions are met: |
| * Redistributions of source code must retain the above copyright |
| notice, this list of conditions and the following disclaimer. |
| * Redistributions in binary form must reproduce the above copyright |
| notice, this list of conditions and the following disclaimer in the |
| documentation and/or other materials provided with the distribution. |
| * Neither the name of the Synthetic Semantics nor the names of its |
| contributors may be used to endorse or promote products derived |
| from this software without specific prior written permission. |
| |
| THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS |
| "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT |
| LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR |
| A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL SYNTHETIC |
| SEMANTICS LLC BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, |
| EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, |
| PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR |
| PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF |
| LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING |
| NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS |
| SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. |
| |
+-----------------------------------------------------------------------------*/
module.exports = {
//-------------------------------------------------------------------
// Timer functions
timerStart: function () {
return new Date().getTime();
},
timerStop: function (timer, nOps, label, myID) {
var now = new Date().getTime();
var opsPerSec = (nOps * 1000000) / ((now - timer) * 1000);
if (typeof myID === undefined || myID === 0) {
console.log(fmtNumber(nOps) + label + fmtNumber(Math.floor(opsPerSec).toString()) + " ops/sec");
}
},
//---------------------------------------------------------------------------
// Generate a random integer within a range (inclusive) from 'low' to 'high'
randomInRange: function (low, high) {
return ( Math.floor((Math.random() * (high - low)) + low) );
}
};
function fmtNumber(n) {
var s = ' ' + n.toString().replace(/\B(?=(\d{3})+(?!\d))/g, ",");
if (n < 1) return n;
else {
return s.substr(s.length - 15, s.length);
}
}
//---------------------------------------------------------------------------
// Return an array with only unique elements
Array.prototype.unique = function () {
return this.filter(function (value, index, self) {
return self.indexOf(value) === index;
});
};
//---------------------------------------------------------------------------
// Randomize the order of data in an array
Array.prototype.shuffle = function () {
for (var j, x, i = this.length; i; j = Math.floor(Math.random() * i), x = this[--i], this[i] = this[j], this[j] = x);
return this;
};
================================================
FILE: Tests/test_alloc.c
================================================
/*-----------------------------------------------------------------------------+
| Extended Memory Semantics (EMS) Version 1.0.0 |
| Synthetic Semantics http://www.synsem.com/ mogill@synsem.com |
+-----------------------------------------------------------------------------+
| Copyright (c) 2011-2014, Synthetic Semantics LLC. All rights reserved. |
| Copyright (c) 2015-2016, Jace A Mogill. All rights reserved. |
| |
| Redistribution and use in source and binary forms, with or without |
| modification, are permitted provided that the following conditions are met: |
| * Redistributions of source code must retain the above copyright |
| notice, this list of conditions and the following disclaimer. |
| * Redistributions in binary form must reproduce the above copyright |
| notice, this list of conditions and the following disclaimer in the |
| documentation and/or other materials provided with the distribution. |
| * Neither the name of the Synthetic Semantics nor the names of its |
| contributors may be used to endorse or promote products derived |
| from this software without specific prior written permission. |
| |
| THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS |
| "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT |
| LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR |
| A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL SYNTHETIC |
| SEMANTICS LLC BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, |
| EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, |
| PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR |
| PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF |
| LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING |
| NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS |
| SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. |
| |
+-----------------------------------------------------------------------------*/
#include "../Addon/ems_alloc.h"
#include
static void
test_size(struct emsMem *b, int64_t addr) {
int64_t s = emsMem_size(b,addr);
printf("size %lld (sz = %lld)\n",addr,s);
}
static int64_t
test_alloc(struct emsMem *b, int64_t sz) {
int64_t r = emsMem_alloc(b,sz);
printf("alloc %lld (sz= %lld)\n",r,sz);
// emsMem_dump(b);
test_size(b, r);
return r;
}
static void
test_free(struct emsMem *b, int64_t addr) {
printf("free %lld\n",addr);
emsMem_free(b,addr);
// emsMem_dump(b);
}
int main() {
#define P(x) printf(">>>>>>>%d %d\n", x, __builtin_ctzl(emsNextPow2(x)))
/*
P(1);
P(10);
P(127);
P(128);
P(129);
exit(1);
*/
struct emsMem * b = emsMem_new(16);
// int64_t zz = test_alloc(b,65536);
// printf("should fail: %lld\n", zz);
int64_t x1 = test_alloc(b,65535);
// int64_t x2 = test_alloc(b,1);
emsMem_dump(b);
test_free(b,x1);
// test_free(b,x2);
emsMem_dump(b);
int64_t m1 = test_alloc(b,4);
test_size(b,m1);
int64_t m2 = test_alloc(b,9);
test_size(b,m2);
int64_t m3 = test_alloc(b,3);
// struct emsMem *arr[100];
int64_t arr[100];
for(int i = 0; i < 50; i ++) {
arr[i] = test_alloc(b, i*2);
}
test_size(b,m3);
int64_t m4 = test_alloc(b,7);
int64_t m5a = test_alloc(b,302);
test_free(b,m3);
test_free(b,m1);
for(int64_t i = 0; i < 50; i ++) {
test_free(b, arr[(i+13)%50]);
}
// test_alloc(b, 1000);
test_free(b,m4);
test_free(b,m2);
int64_t m5 = test_alloc(b,32);
test_free(b,m5);
int64_t m6 = test_alloc(b,0);
test_free(b,m6);
test_free(b,m5a);
emsMem_dump(b);
emsMem_delete(b);
return 0;
}
================================================
FILE: Tests/tm.js
================================================
/*-----------------------------------------------------------------------------+
| Extended Memory Semantics (EMS) Version 1.0.0 |
| Synthetic Semantics http://www.synsem.com/ mogill@synsem.com |
+-----------------------------------------------------------------------------+
| Copyright (c) 2011-2014, Synthetic Semantics LLC. All rights reserved. |
| Copyright (c) 2015-2016, Jace A Mogill. All rights reserved. |
| |
| Redistribution and use in source and binary forms, with or without |
| modification, are permitted provided that the following conditions are met: |
| * Redistributions of source code must retain the above copyright |
| notice, this list of conditions and the following disclaimer. |
| * Redistributions in binary form must reproduce the above copyright |
| notice, this list of conditions and the following disclaimer in the |
| documentation and/or other materials provided with the distribution. |
| * Neither the name of the Synthetic Semantics nor the names of its |
| contributors may be used to endorse or promote products derived |
| from this software without specific prior written permission. |
| |
| THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS |
| "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT |
| LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR |
| A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL SYNTHETIC |
| SEMANTICS LLC BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, |
| EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, |
| PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR |
| PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF |
| LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING |
| NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS |
| SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. |
| |
+-----------------------------------------------------------------------------*/
var ems = require('ems')(parseInt(process.argv[2]));
var util = require('./testUtils');
var assert = require('assert');
var arrLen = 1000000;
var nTransactions = 1000000;
var nTables = 6;
var maxNops = 5;
var tables = [];
var workQ = ems.new({
dimensions: [nTransactions + ems.nThreads],
heapSize: nTransactions * 200,
useExisting: false
});
var totalNops = ems.new(2);
var checkNops = ems.new(1);
var startTime = util.timerStart();
//---------------------------------------------------------------------------
// Create all the tables
for (var tableN = 0; tableN < nTables; tableN++) {
tables[tableN] = ems.new({
dimensions: [arrLen],
heapSize: 0,
useExisting: false,
filename: '/tmp/EMS_tm' + tableN,
dataFill: 0,
doDataFill: true
});
}
util.timerStop(startTime, nTables, " tables initialized ", ems.myID);
//---------------------------------------------------------------------------
// One thread enqueues all the work
ems.master(function () {
totalNops.writeXF(0, 0);
totalNops.writeXF(1, 0);
checkNops.writeXF(0, 0);
});
ems.barrier();
startTime = util.timerStart();
//---------------------------------------------------------------------------
// Generate operations involving random elements in random EMS arrays
// and enqueue them on the work queue
ems.parForEach(0, nTransactions, function (transN) {
var ops = [];
var nOps = util.randomInRange(1, maxNops);
for (var opN = 0; opN < nOps; opN++) {
var tableN = util.randomInRange(0, nTables);
var idx = util.randomInRange(0, arrLen);
if (transN % 2 == 0 || opN % 3 > 0) {
ops.push([tableN, idx, true]);
}
// if(opN % 3 > 0) { ops.push([tableN, idx, true]) }
else {
ops.push([tableN, idx]);
}
}
// De-duplicate operations in a transaction which would deadlock
var indicies = [];
var uids = [];
for (var i = 0; i < ops.length; i++) {
indicies[i] = i;
uids[i] = (ops[i][0] * 1000000000000) + ops[i][1];
}
var uniq = [];
for (opN = 0; opN < ops.length; opN++) {
var isDupl = false;
for (var checkN = 0; checkN < ops.length; checkN++) {
if (opN != checkN && uids[opN] == uids[checkN]) {
isDupl = true;
break;
}
}
if (!isDupl) {
uniq.push(ops[opN]);
}
}
workQ.enqueue(JSON.stringify(uniq));
});
// After all the work has been enqueued, add DONE semaphores to the
// end of the queue so they are processed only after all the work
// has been issued. Each thread enqueues one event and can only
// consume one before exiting.
workQ.enqueue("DONE");
util.timerStop(startTime, nTransactions, " transactions enqueued ", ems.myID);
var rwNops = 0;
var readNops = 0;
startTime = util.timerStart();
while (true) {
var str = workQ.dequeue();
if (str !== undefined) {
if (str === "DONE") {
break;
} else {
var ops = JSON.parse(str);
for (var opN = 0; opN < ops.length; opN++) {
ops[opN][0] = tables[ops[opN][0]];
}
var transaction = ems.tmStart(ops);
ops.forEach(function (op) {
var tmp = op[0].read(op[1]);
if (op[2] != true) {
rwNops++;
op[0].write(op[1], tmp + 1);
} else {
readNops++;
}
});
ems.tmEnd(transaction, true);
}
} else {
ems.diag("Should not find nothing in queue");
}
}
totalNops.faa(0, rwNops);
totalNops.faa(1, readNops);
ems.barrier();
util.timerStop(startTime, nTransactions, " transactions performed", ems.myID);
util.timerStop(startTime, totalNops.readFF(0), " table updates ", ems.myID);
util.timerStop(startTime, totalNops.readFF(0) + totalNops.readFF(1), " elements referenced ", ems.myID);
startTime = util.timerStart();
ems.parForEach(0, nTables, function (tableN) {
var localSum = 0;
for (var idx = 0; idx < arrLen; idx++) {
localSum += tables[tableN].read(idx);
}
checkNops.faa(0, localSum);
}, 'dynamic');
util.timerStop(startTime, nTables * arrLen, " elements checked ", ems.myID);
ems.master(function () {
assert(checkNops.readFF(0) === totalNops.readFF(0),
"Error in final sum = " + checkNops.readFF(0) + " should be=" + totalNops.readFF(0));
});
================================================
FILE: Tests/tm_noq.js
================================================
/*-----------------------------------------------------------------------------+
| Extended Memory Semantics (EMS) Version 1.3.0 |
| Synthetic Semantics http://www.synsem.com/ mogill@synsem.com |
+-----------------------------------------------------------------------------+
| Copyright (c) 2011-2014, Synthetic Semantics LLC. All rights reserved. |
| Copyright (c) 2015-2016, Jace A Mogill. All rights reserved. |
| |
| Redistribution and use in source and binary forms, with or without |
| modification, are permitted provided that the following conditions are met: |
| * Redistributions of source code must retain the above copyright |
| notice, this list of conditions and the following disclaimer. |
| * Redistributions in binary form must reproduce the above copyright |
| notice, this list of conditions and the following disclaimer in the |
| documentation and/or other materials provided with the distribution. |
| * Neither the name of the Synthetic Semantics nor the names of its |
| contributors may be used to endorse or promote products derived |
| from this software without specific prior written permission. |
| |
| THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS |
| "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT |
| LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR |
| A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL SYNTHETIC |
| SEMANTICS LLC BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, |
| EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, |
| PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR |
| PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF |
| LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING |
| NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS |
| SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. |
| |
+-----------------------------------------------------------------------------*/
var ems = require('ems')(parseInt(process.argv[2]));
var util = require('./testUtils');
var assert = require('assert');
var arrLen = 1000000;
var nTransactions = 400000;
var nTables = 6;
var maxNops = 5;
var tables = [];
var totalNops = ems.new(2);
var checkNops = ems.new(1);
//---------------------------------------------------------------------------
// Create all the tables
for (var tableN = 0; tableN < nTables; tableN++) {
tables[tableN] = ems.new({
dimensions: [arrLen],
heapSize: 0,
useExisting: false,
filename: '/tmp/EMS_tm' + tableN,
dataFill: 0,
doDataFill: true,
setFEtags: 'full'
});
}
// One thread enqueues all the work
ems.master(function () {
totalNops.writeXF(0, 0);
totalNops.writeXF(1, 0);
checkNops.writeXF(0, 0);
});
ems.barrier();
function makeWork() {
var ops = [];
var nOps = util.randomInRange(1, maxNops);
for (var opN = 0; opN < nOps; opN++) {
var tableN = util.randomInRange(0, nTables);
var idx = util.randomInRange(0, arrLen);
if (idx % 10 < 5 || opN % 3 > 0) {
ops.push([tableN, idx, true]);
} else {
ops.push([tableN, idx]);
}
}
// De-duplicate operations in a transaction which would deadlock
var indicies = [];
var uids = [];
for (var i = 0; i < ops.length; ++i) {
indicies[i] = i;
uids[i] = (ops[i][0] * 1000000000000) + ops[i][1];
}
var uniq = [];
for (opN = 0; opN < ops.length; opN++) {
var isDupl = false;
for (var checkN = 0; checkN < ops.length; checkN++) {
if (opN != checkN && uids[opN] == uids[checkN]) {
isDupl = true;
break;
}
}
if (!isDupl) {
uniq.push(ops[opN]);
}
}
return (JSON.stringify(uniq));
}
var rwNops = 0;
var readNops = 0;
var startTime = util.timerStart();
ems.parForEach(0, nTransactions, function (iter) {
var str = makeWork(iter);
if (str !== undefined) {
var ops = JSON.parse(str);
for (var opN = 0; opN < ops.length; opN++) {
ops[opN][0] = tables[ops[opN][0]];
}
var transaction = ems.tmStart(ops);
ops.forEach(function (op) {
var tmp = op[0].read(op[1]);
if (op[2] != true) {
rwNops++;
op[0].write(op[1], tmp + 1);
} else {
readNops++;
}
});
ems.tmEnd(transaction, true);
}
});
totalNops.faa(0, rwNops);
totalNops.faa(1, readNops);
ems.barrier();
util.timerStop(startTime, nTransactions, " transactions performed ", ems.myID);
util.timerStop(startTime, totalNops.readFF(0), " table updates ", ems.myID);
util.timerStop(startTime, totalNops.readFF(0) + totalNops.readFF(1), " elements referenced ", ems.myID);
startTime = util.timerStart();
ems.parForEach(0, nTables, function (tableN) {
var localSum = 0;
for (var idx = 0; idx < arrLen; idx++) {
localSum += tables[tableN].read(idx);
}
checkNops.faa(0, localSum);
}, 'dynamic');
util.timerStop(startTime, nTables * arrLen, " elements checked ", ems.myID);
ems.master(function () {
assert(checkNops.readFF(0) == totalNops.readFF(0),
"Error in final sum = " + checkNops.readFF(0) + " should be=" + totalNops.readFF(0));
});
================================================
FILE: Tests/tm_noq_strings.js
================================================
/*-----------------------------------------------------------------------------+
| Extended Memory Semantics (EMS) Version 1.4.0 |
| Synthetic Semantics http://www.synsem.com/ mogill@synsem.com |
+-----------------------------------------------------------------------------+
| Copyright (c) 2011-2014, Synthetic Semantics LLC. All rights reserved. |
| Copyright (c) 2015-2016, Jace A Mogill. All rights reserved. |
| |
| Redistribution and use in source and binary forms, with or without |
| modification, are permitted provided that the following conditions are met: |
| * Redistributions of source code must retain the above copyright |
| notice, this list of conditions and the following disclaimer. |
| * Redistributions in binary form must reproduce the above copyright |
| notice, this list of conditions and the following disclaimer in the |
| documentation and/or other materials provided with the distribution. |
| * Neither the name of the Synthetic Semantics nor the names of its |
| contributors may be used to endorse or promote products derived |
| from this software without specific prior written permission. |
| |
| THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS |
| "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT |
| LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR |
| A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL SYNTHETIC |
| SEMANTICS LLC BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, |
| EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, |
| PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR |
| PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF |
| LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING |
| NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS |
| SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. |
| |
+-----------------------------------------------------------------------------*/
var ems = require('ems')(parseInt(process.argv[2]));
var util = require('./testUtils');
var arrLen = 1000000;
var nTransactions = 4000000;
var nTables = 6;
var maxNops = 5;
var tables = [];
var totalNops = ems.new(2);
var checkNops = ems.new(1);
var tableN;
//---------------------------------------------------------------------------
// Create all the tables
for (tableN = 0; tableN < nTables; tableN++) {
tables[tableN] = ems.new({
dimensions: [arrLen],
heapSize: arrLen * 40,
useExisting: false,
filename: '/tmp/EMS_tm' + tableN,
dataFill: 0,
doDataFill: true
});
}
// One thread enqueues all the work
ems.master(function () {
totalNops.writeXF(0, 0);
totalNops.writeXF(1, 0);
checkNops.writeXF(0, 0);
});
ems.barrier();
function makeWork() {
var ops = [];
var nOps = util.randomInRange(1, maxNops);
// var indexes = []
for (var opN = 0; opN < nOps; opN++) {
var tableN = util.randomInRange(0, nTables);
var idx = util.randomInRange(0, arrLen);
if (idx % 10 < 5 || opN % 3 > 0) {
ops.push([tableN, idx, true]);
}
else {
ops.push([tableN, idx]);
}
}
// De-duplicate operations in a transaction which would deadlock
var indicies = [];
var uids = [];
for (var i = 0; i < ops.length; ++i) {
indicies[i] = i;
uids[i] = (ops[i][0] * 1000000000000) + ops[i][1];
}
var uniq = [];
for (opN = 0; opN < ops.length; opN++) {
var isDupl = false;
for (var checkN = 0; checkN < ops.length; checkN++) {
if (opN != checkN && uids[opN] == uids[checkN]) {
isDupl = true;
break;
}
}
if (!isDupl) {
uniq.push(ops[opN]);
}
}
return (JSON.stringify(uniq));
}
var rwNops = 0;
var readNops = 0;
var startTime = util.timerStart();
//---------------------------------------------------------------------------
// Generate operations involving random elements in random EMS arrays
// and enqueue them on the work queue
//
ems.parForEach(0, nTransactions, function (iter) {
var str = makeWork(iter);
if (str !== undefined) {
var ops = JSON.parse(str);
for (var opN = 0; opN < ops.length; opN++) {
ops[opN][0] = tables[ops[opN][0]];
}
var transaction = ems.tmStart(ops);
ops.forEach(function (op) {
var tmp = op[0].read(op[1]);
if (op[2] != true) {
rwNops++;
op[0].write(op[1], tmp + ' ' + 1);
} else {
readNops++;
}
});
ems.tmEnd(transaction, true);
}
});
totalNops.faa(0, rwNops);
totalNops.faa(1, readNops);
ems.barrier();
util.timerStop(startTime, nTransactions, " transactions performed ", ems.myID);
util.timerStop(startTime, totalNops.readFF(0), " table updates ", ems.myID);
util.timerStop(startTime, totalNops.readFF(0) + totalNops.readFF(1), " elements referenced ", ems.myID);
startTime = util.timerStart();
if (true) {
ems.parForEach(0, nTables, function (tableN) {
// var localSum = 0
for (var idx = 0; idx < arrLen; idx++) {
tables[tableN].read(idx);
}
}, 'dynamic');
} else {
for (tableN = 0; tableN < nTables; tableN++) {
ems.parForEach(0, arrLen, function (idx) {
checkNops.faa(0, tables[tableN].read(idx));
});
}
}
util.timerStop(startTime, nTables * arrLen, " elements checked ", ems.myID);
================================================
FILE: Tests/v8Types.js
================================================
/*-----------------------------------------------------------------------------+
| Extended Memory Semantics (EMS) Version 1.3.0 |
| Synthetic Semantics http://www.synsem.com/ mogill@synsem.com |
+-----------------------------------------------------------------------------+
| Copyright (c) 2011-2014, Synthetic Semantics LLC. All rights reserved. |
| Copyright (c) 2015-2016, Jace A Mogill. All rights reserved. |
| |
| Redistribution and use in source and binary forms, with or without |
| modification, are permitted provided that the following conditions are met: |
| * Redistributions of source code must retain the above copyright |
| notice, this list of conditions and the following disclaimer. |
| * Redistributions in binary form must reproduce the above copyright |
| notice, this list of conditions and the following disclaimer in the |
| documentation and/or other materials provided with the distribution. |
| * Neither the name of the Synthetic Semantics nor the names of its |
| contributors may be used to endorse or promote products derived |
| from this software without specific prior written permission. |
| |
| THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS |
| "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT |
| LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR |
| A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL SYNTHETIC |
| SEMANTICS LLC BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, |
| EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, |
| PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR |
| PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF |
| LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING |
| NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS |
| SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. |
| |
+-----------------------------------------------------------------------------*/
'use strict';
var ems = require('ems')(parseInt(process.argv[2]), false);
var assert = require('assert');
var arrLen = 10000;
var a = ems.new(arrLen, arrLen * 400);
var arrayElem = ['abcd', true, 1234.567, false, {x: 'xxx', y: 'yyyyy'}, 987, null, [10, 11, 12, 13]];
var objElem = {a: 1, b: 321.653, c: 'asdasd'};
var objMap = ems.new({
dimensions: [arrLen],
heapSize: arrLen * 200,
useMap: true,
useExisting: false,
setFEtags: 'full'
});
var arrObj = ems.new({
dimensions: [arrLen],
heapSize: arrLen * 200,
useMap: false,
useExisting: false,
setFEtags: 'full',
dataFill: arrayElem,
doDataFill: true
});
if (ems.myID == 0) {
objMap.writeXF('any obj', objElem);
assert(objMap.readFF('any obj').a === objElem.a);
assert(objMap.readFF('any obj').b === objElem.b);
assert(objMap.readFE('any obj').c === objElem.c);
arrayElem.forEach(function (elem, idx) {
if (typeof elem === 'object') {
assert(typeof arrObj.readFF(123)[idx] === 'object');
} else {
var readback = arrObj.readFF(123);
assert(readback[idx] === elem);
}
});
arrObj.readFE(123);
var newObj = {xa: 10000, xb: 32100.653, xc: 'xxxxxxxasdasd'};
objMap.writeEF('any obj', newObj);
assert(objMap.readFF('any obj').xa === newObj.xa);
assert(objMap.readFF('any obj').xb === newObj.xb);
assert(objMap.readFE('any obj').xc === newObj.xc);
arrObj.writeEF(123, 'overwrite the old array');
arrObj.writeXF(1, []);
var newArr = [9, 8, 7, 6, , , , 'abs', {one: 1, two: 2, three: 'threeeeee'}, 1, 2, 3];
arrObj.writeXF(2, newArr);
assert(arrObj.readFE(123) === 'overwrite the old array');
assert(arrObj.readFE(1).length === 0);
newArr.forEach(function (elem, idx) {
if (typeof elem == 'object') {
if (typeof arrObj.readFF(2)[idx] != 'object') {
console.log('Object in array is no longer an object?');
}
} else {
assert(arrObj.readFF(2)[idx] === elem);
}
});
var newerObj = {q: 123, r: 345, x: [1, 2, 3, 4]};
arrObj.writeEF(123, newerObj);
assert(arrObj.readFF(123).q === newerObj.q);
assert(arrObj.readFF(123).r === newerObj.r);
assert(arrObj.readFF(123).x[2] === newerObj.x[2]);
}
ems.barrier();
//----------------------------------------
var newIdx, newVal, oldVal, js;
var data = [false, true, 1234, 987.654321, 'hello', undefined];
for (var old = 0; old < data.length; old++) {
for (newIdx = 0; newIdx < data.length; newIdx++) {
a.writeXF(ems.myID, data[old]);
js = data[old];
js += data[newIdx];
oldVal = a.faa(ems.myID, data[newIdx]);
newVal = a.readFF(ems.myID);
assert(((newVal === js) || (isNaN(oldVal) && isNaN(newVal) ) ||
(isNaN(newVal) && data[newIdx] === undefined) ),
'FAA: old=' + data[old] + ' new=' + data[newIdx] +
' oldVal=' + oldVal + '/' + typeof oldVal + ' newVal=' +
newVal + '/' + typeof newVal + ' js=' + js + '/' + typeof js);
}
}
ems.barrier();
var id = (ems.myID + 1) % ems.nThreads;
for (var memIdx = 0; memIdx < data.length; memIdx++) {
for (var oldIdx = 0; oldIdx < data.length; oldIdx++) {
for (newIdx = 0; newIdx < data.length; newIdx++) {
a.writeXF(id, data[memIdx]);
var memVal = a.cas(id, data[oldIdx], data[newIdx]);
newVal = a.readFF(id);
js = data[memIdx];
if (js === data[oldIdx]) {
js = data[newIdx];
}
assert(js === newVal,
'CAS: intial=' + data[memIdx] + " memval=" + memVal +
" readback=" + newVal + " oldVal=" + data[oldIdx] +
' expected=' + data[newIdx] + ' JS:' + js);
}
}
}
ems.parForEach(0, arrLen, function (idx) {
a.writeXF(idx, undefined);
a.faa(idx, undefined);
a.faa(idx, 'bye byte');
a.faa(idx, ems.myID);
a.faa(idx, 0.1);
a.faa(idx, false);
assert(a.readFF(idx) == 'nanbye byte' + ems.myID + '0.100000false',
'Failed match =' + a.read(idx));
});
//-----------------------------------------------------------
ems.parForEach(0, arrLen, function (idx) {
a.writeXF(idx, 0);
});
var nTimes = 150;
ems.parForEach(0, arrLen, function (previdx) {
for (var i = 0; i < nTimes; i++) {
a.faa((previdx + i) % arrLen, 1)
}
});
ems.parForEach(0, arrLen, function (idx) {
a.faa(idx, 0.1);
a.faa(idx, 'fun!');
a.faa(idx, false);
assert(a.readFE(idx) == nTimes + '.100000fun!false',
'Failed match =' + a.read(idx) + ' idx=' + idx);
});
================================================
FILE: Vagrantfile
================================================
# -*- mode: ruby -*-
# vi: set ft=ruby :
# All Vagrant configuration is done below. The "2" in Vagrant.configure
# configures the configuration version (we support older styles for
# backwards compatibility). Please don't change it unless you know what
# you're doing.
Vagrant.configure(2) do |config|
# The most common configuration options are documented and commented below.
# For a complete reference, please see the online documentation at
# https://docs.vagrantup.com.
# Every Vagrant development environment requires a box. You can search for
# boxes at https://atlas.hashicorp.com/search.
config.vm.box = "ubuntu/trusty64"
# Disable automatic box update checking. If you disable this, then
# boxes will only be checked for updates when the user runs
# `vagrant box outdated`. This is not recommended.
# config.vm.box_check_update = false
# Create a forwarded port mapping which allows access to a specific port
# within the machine from a port on the host machine. In the example below,
# accessing "localhost:8080" will access port 80 on the guest machine.
# config.vm.network "forwarded_port", guest: 80, host: 8080
# Create a private network, which allows host-only access to the machine
# using a specific IP.
# config.vm.network "private_network", ip: "192.168.33.10"
# Create a public network, which generally matched to bridged network.
# Bridged networks make the machine appear as another physical device on
# your network.
# config.vm.network "public_network"
# Share an additional folder to the guest VM. The first argument is
# the path on the host to the actual folder. The second argument is
# the path on the guest to mount the folder. And the optional third
# argument is a set of non-required options.
# config.vm.synced_folder "../data", "/vagrant_data"
# Provider-specific configuration so you can fine-tune various
# backing providers for Vagrant. These expose provider-specific options.
# Example for VirtualBox:
#
# config.vm.provider "virtualbox" do |vb|
# # Display the VirtualBox GUI when booting the machine
# vb.gui = true
#
# # Customize the amount of memory on the VM:
# vb.memory = 1024
# vb.cpus = 1
# end
#
# View the documentation for the provider you are using for more
# information on available options.
# Define a Vagrant Push strategy for pushing to Atlas. Other push strategies
# such as FTP and Heroku are also available. See the documentation at
# https://docs.vagrantup.com/v2/push/atlas.html for more information.
# config.push.define "atlas" do |push|
# push.app = "YOUR_ATLAS_USERNAME/YOUR_APPLICATION_NAME"
# end
# Enable provisioning with a shell script. Additional provisioners such as
# Puppet, Chef, Ansible, Salt, and Docker are also available. Please see the
# documentation for more information about their specific syntax and use.
# config.vm.provision "shell", inline: <<-SHELL
# sudo apt-get update
# sudo apt-get install -y apache2
# SHELL
config.vm.provision "shell", inline: <<-SHELL
sudo apt-get update
sudo curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.35.3/install.sh | bash
SHELL
end
================================================
FILE: binding.gyp
================================================
{
"targets": [
{
"target_name": "ems",
"sources": [
"src/collectives.cc", "src/ems.cc", "src/ems_alloc.cc", "src/loops.cc",
"nodejs/nodejs.cc", "src/primitives.cc", "src/rmw.cc"],
'include_dirs': ["
================================================
FILE: nodejs/ems.js
================================================
/*-----------------------------------------------------------------------------+
| Extended Memory Semantics (EMS) Version 1.6.1 |
| http://mogill.com/ jace@mogill.com |
+-----------------------------------------------------------------------------+
| Copyright (c) 2011-2014, Synthetic Semantics LLC. All rights reserved. |
| Copyright (c) 2019 Aleksander J Budzynowski. Update NAN to N-API. |
| Copyright (c) 2015-2020, Jace A Mogill. All rights reserved. |
| |
| Redistribution and use in source and binary forms, with or without |
| modification, are permitted provided that the following conditions are met: |
| * Redistributions of source code must retain the above copyright |
| notice, this list of conditions and the following disclaimer. |
| * Redistributions in binary form must reproduce the above copyright |
| notice, this list of conditions and the following disclaimer in the |
| documentation and/or other materials provided with the distribution. |
| * Neither the name of the Synthetic Semantics nor the names of its |
| contributors may be used to endorse or promote products derived |
| from this software without specific prior written permission. |
| |
| THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS |
| "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT |
| LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR |
| A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL SYNTHETIC |
| SEMANTICS LLC BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, |
| EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, |
| PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR |
| PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF |
| LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING |
| NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS |
| SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. |
| |
+-----------------------------------------------------------------------------*/
"use strict";
var fs = require("fs");
var child_process = require("child_process");
var EMS = require("bindings")("ems.node");
var EMSglobal;
// The Proxy object is built in or defined by Reflect
try {
var EMS_Harmony_Reflect = require("harmony-reflect");
} catch (err) {
// Not installed, but possibly not needed anyhow
}
//==================================================================
// Convert the different possible index types into a linear EMS index
function EMSidx(indexes, // Index given by application
EMSarray) { // EMS array being indexed
var idx = 0;
if (indexes instanceof Array) { // Is a Multidimension array: [x,y,z]
indexes.forEach(function (x, i) {
idx += x * EMSarray.dimStride[i];
});
} else {
if (!(typeof indexes === "number") && !EMSarray.useMap) { // If no map, only use integers
console.log("EMS ERROR: Non-integer index used, but EMS memory was not configured to use a map (useMap)",
indexes, typeof indexes, EMSarray.useMap);
idx = -1;
} else { // Is a mappable intrinsic type
idx = indexes;
}
}
return idx;
}
//==================================================================
// Print a message to the console with a prefix indicating which
// thread is printing
function EMSdiag(text) {
console.log("EMStask " + this.myID + ": " + text);
}
//==================================================================
// Co-Begin a parallel region, executing the function "func"
function EMSparallel() {
EMSglobal.inParallelContext = true;
var user_args = (arguments.length === 1?[arguments[0]]:Array.apply(null, arguments));
var func = user_args.pop(); // Remove the function
// Loop over remote processes, starting each of them
this.tasks.forEach(function (task, taskN) {
task.send({"taskN": taskN + 1, "args": user_args, "func": func.toString()});
task.send({"taskN": taskN + 1, "args": [], "func": "function() { ems.barrier(); }"});
});
func.apply(null, user_args); // Invoke on master process
EMSbarrier(); // Wait for all processes to finish
EMSglobal.inParallelContext = false;
}
//==================================================================
// Execute the local iterations of a decomposed loop with
// the specified scheduling.
function EMSparForEach(start, // First iteration's index
end, // Final iteration's index
loopBody, // Function to execute each iteration
scheduleType, // Load balancing technique
minChunk) { // Smallest block of iterations
var sched = scheduleType;
var idx;
if (typeof sched === "undefined") {
sched = "guided";
}
if (typeof minChunk === "undefined") {
minChunk = 1;
}
switch (scheduleType) {
case "static": // Evenly divide the iterations across threads
var range = end - start;
var blocksz = Math.floor(range / EMSglobal.nThreads) + 1;
var s = blocksz * EMSglobal.myID;
var e = blocksz * (EMSglobal.myID + 1);
if (e > end) {
e = end;
}
for (idx = s; idx < e; idx += 1) {
loopBody(idx + start);
}
break;
case "dynamic":
case "guided":
default:
// Initialize loop bounds, block size, etc.
EMSglobal.loopInit(start, end, sched, minChunk);
// Do not enter loop until all threads have completed initialization
// If the barrier is present at the loop end, this may replaced
// with first-thread initialization.
var extents;
EMSbarrier();
do {
extents = EMSglobal.loopChunk();
for (idx = extents.start; idx < extents.end; idx++) {
loopBody(idx);
}
} while (extents.end - extents.start > 0);
}
// Do not proceed until all iterations have completed
// TODO: OpenMP offers NOWAIT to skip this barrier
EMSbarrier();
}
//==================================================================
// Start a Transaction
// Input is an array containing EMS elements to transition from
// Full to Empty:
// arr = [ [ emsArr0, idx0 ], [ emsArr1, idx1, true ], [ emsArr2, idx2 ] ]
function EMStmStart(emsElems) {
var MAX_ELEM_PER_REGION = 10000000000000;
// Build a list of indexes of the elements to lock which will hold
// the elements sorted for deadlock free acquisition
var indicies = [];
for (var i = 0; i < emsElems.length; ++i) indicies[i] = i;
// Sort the elements to lock according to a global ordering
var sorted = indicies.sort(function (a, b) {
return ( ((emsElems[a][0].regionN * MAX_ELEM_PER_REGION) + emsElems[a][1]) -
((emsElems[b][0].regionN * MAX_ELEM_PER_REGION) + emsElems[b][1]) )
});
// Acquire the locks in the deadlock free order, saving the contents
// of the memory when it locked.
// Mark read-write data as Empty, and read-only data under a readers-writer lock
var tmHandle = [];
sorted.forEach(function (e) {
var val;
if (emsElems[e][2] === true) {
val = emsElems[e][0].readRW(emsElems[e][1]);
} else {
val = emsElems[e][0].readFE(emsElems[e][1]);
}
tmHandle.push([emsElems[e][0], emsElems[e][1], emsElems[e][2], val])
});
return tmHandle;
}
//==================================================================
// Commit or abort a transaction
// The tmHandle contains the result from tmStart:
// [ [ EMSarray, index, isReadOnly, origValue ], ... ]
function EMStmEnd(tmHandle, // The returned value from tmStart
doCommit) { // Commit or Abort the transaction
tmHandle.forEach(function (emsElem) {
if (emsElem[2] === true) {
// Is a read-only element
emsElem[0].releaseRW(emsElem[1]);
} else {
if (doCommit) {
// Is read-write, keep current value and mark Full
emsElem[0].setTag(emsElem[1], true);
} else {
// Write back the original value and mark full
emsElem[0].writeEF(emsElem[1], emsElem[3]);
}
}
});
}
//==================================================================
function EMSreturnData(value) {
if (typeof value === "object") {
var retval;
try {
if (value.data[0] === "[" && value.data.slice(-1) === "]") {
retval = eval(value.data);
} else {
retval = JSON.parse(value.data);
}
} catch (err) {
// TODO: Throw error?
console.log('EMSreturnData: Corrupt memory, unable to reconstruct JSON value of (' +
value.data + ')\n');
retval = undefined;
}
return retval;
} else {
return value;
}
}
//==================================================================
// Synchronize memory with storage
//
function EMSsync(emsArr) {
return this.data.sync(emsArr);
}
//==================================================================
// Convert an EMS index into a mapped key
function EMSindex2key(index) {
if(typeof(index) !== "number") {
console.log('EMSindex2key: Index (' + index + ') is not an integer');
return undefined;
}
return this.data.index2key(index);
}
//==================================================================
// Wrappers around Stacks and Queues
function EMSpush(value) {
if (typeof value === "object") {
return this.data.push(JSON.stringify(value), true);
} else {
return this.data.push(value);
}
}
function EMSpop() {
return EMSreturnData(this.data.pop());
}
function EMSdequeue() {
return EMSreturnData(this.data.dequeue());
}
function EMSenqueue(value) {
if (typeof value === "object") {
return this.data.enqueue(JSON.stringify(value), true); // Retuns only integers
} else {
return this.data.enqueue(value); // Retuns only integers
}
}
//==================================================================
// Wrappers around Primitive AMOs
// Translate EMS maps and multi-dimensional array indexes/keys
// into EMS linear addresses
// Apparently it is illegal to pass a native function as an argument
function EMSwrite(indexes, value) {
var linearIndex = EMSidx(indexes, this);
if (typeof value === "object") {
this.data.write(linearIndex, JSON.stringify(value), true);
} else {
this.data.write(linearIndex, value);
}
}
function EMSwriteEF(indexes, value) {
var linearIndex = EMSidx(indexes, this);
if (typeof value === "object") {
this.data.writeEF(linearIndex, JSON.stringify(value), true);
} else {
this.data.writeEF(linearIndex, value);
}
}
function EMSwriteXF(indexes, value) {
var linearIndex = EMSidx(indexes, this);
if (typeof value === "object") {
this.data.writeXF(linearIndex, JSON.stringify(value), true);
} else {
this.data.writeXF(linearIndex, value);
}
}
function EMSwriteXE(indexes, value) {
var nativeIndex = EMSidx(indexes, this);
if (typeof value === "object") {
this.data.writeXE(nativeIndex, JSON.stringify(value), true);
} else {
this.data.writeXE(nativeIndex, value);
}
}
function EMSread(indexes) {
return EMSreturnData(this.data.read(EMSidx(indexes, this)))
}
function EMSreadFE(indexes) {
return EMSreturnData(this.data.readFE(EMSidx(indexes, this)))
}
function EMSreadFF(indexes) {
return EMSreturnData(this.data.readFF(EMSidx(indexes, this)))
}
function EMSreadRW(indexes) {
return EMSreturnData(this.data.readRW(EMSidx(indexes, this)))
}
function EMSreleaseRW(indexes) {
return this.data.releaseRW(EMSidx(indexes, this))
}
function EMSsetTag(indexes, fe) {
return this.data.setTag(EMSidx(indexes, this), fe)
}
function EMSfaa(indexes, val) {
if (typeof val === "object") {
console.log("EMSfaa: Cannot add an object to something");
return undefined;
} else {
return this.data.faa(EMSidx(indexes, this), val); // No returnData(), FAA can only return JSON primitives
}
}
function EMScas(indexes, oldVal, newVal) {
if (typeof newVal === "object") {
console.log("EMScas: ERROR -- objects are not a valid new type");
return undefined;
} else {
return this.data.cas(EMSidx(indexes, this), oldVal, newVal);
}
}
//==================================================================
// Serialize execution through this function
function EMScritical(func, timeout) {
if (typeof timeout === "undefined") {
timeout = 500000; // TODO: Magic number -- long enough for errors, not load imbalance
}
EMSglobal.criticalEnter(timeout);
var retObj = func();
EMSglobal.criticalExit();
return retObj
}
//==================================================================
// Perform func only on thread 0
function EMSmaster(func) {
if (this.myID === 0) {
return func();
}
}
//==================================================================
// Perform the function func once by the first thread to reach
// the function. The final barrier is required because a
// thread may try to execute the next single-execution region
// before other threads have finished this region, which the EMS
// runtime cannot tell apart. Barriers are phased, so a barrier
// is used to prevent any thread from entering the next single-
// execution region before this one is complete
function EMSsingle(func) {
var retObj;
if (this.singleTask()) {
retObj = func();
}
EMSbarrier();
return retObj
}
//==================================================================
// Wrapper around the EMS global barrier
function EMSbarrier(timeout) {
if (EMSglobal.inParallelContext) {
if(typeof timeout === "undefined") {
timeout = 500000; // TODO: Magic number -- long enough for errors, not load imbalance
}
var remaining_time = EMS.barrier.call(EMSglobal, timeout);
if (remaining_time < 0) {
console.log("EMSbarrier: ERROR -- Barrier timed out after", timeout, "iterations.");
// TODO: Probably should throw an error
}
return remaining_time;
}
return timeout;
}
//==================================================================
// Utility functions for determining types
function EMSisArray(a) {
return typeof a.pop !== "undefined"
}
function EMSisObject(o) {
return typeof o === "object" && !EMSisArray(o)
}
function EMSisDefined(x) {
return typeof x !== "undefined"
}
//==================================================================
// Release all resources associated with an EMS memory region
function EMSdestroy(unlink_file) {
EMSbarrier();
if (EMSglobal.myID == 0) {
this.data.destroy(unlink_file);
}
EMSbarrier();
}
//==================================================================
// Creating a new EMS memory region
function EMSnew(arg0, // Maximum number of elements the EMS region can hold
heapSize, // #bytes of memory reserved for strings/arrays/objs/maps/etc
filename // Optional filename for persistent EMS memory
) {
var fillIsJSON = false;
var emsDescriptor = { // Internal EMS descriptor
nElements: 1, // Required: Maximum number of elements in array
heapSize: 0, // Optional, default=0: Space, in bytes, for strings, maps, objects, etc.
mlock: 0, // Optional, 0-100% of EMS memory into RAM
useMap: false, // Optional, default=false: Use a map from keys to indexes
useExisting: false, // Optional, default=false: Preserve data if a file already exists
ES6proxies: false, // Optional, default=false: Inferred EMS read/write syntax
persist: true, // Optional, default=true: Preserve the file after threads exit
doDataFill: false, // Optional, default=false: Data values should be initialized
dataFill: undefined,//Optional, default=false: Value to initialize data to
doSetFEtags: false, // Optional, initialize full/empty tags
setFEtagsFull: true, // Optional, used only if doSetFEtags is true
dimStride: [] // Stride factors for each dimension of multidimensional arrays
};
if (!EMSisDefined(arg0)) { // Nothing passed in, assume length 1
emsDescriptor.dimensions = [1];
} else {
if (EMSisObject(arg0)) { // User passed in emsArrayDescriptor
if (typeof arg0.dimensions !== "undefined") {
if (typeof arg0.dimensions !== "object") {
emsDescriptor.dimensions = [ arg0.dimensions ]
} else {
emsDescriptor.dimensions = arg0.dimensions
}
}
if (typeof arg0.ES6proxies !== "undefined") {
emsDescriptor.ES6proxies = arg0.ES6proxies;
}
if (typeof arg0.heapSize !== "undefined") {
emsDescriptor.heapSize = arg0.heapSize;
}
if (typeof arg0.mlock !== "undefined") {
emsDescriptor.mlock = arg0.mlock
}
if (typeof arg0.useMap !== "undefined") {
emsDescriptor.useMap = arg0.useMap
}
if (typeof arg0.filename !== "undefined") {
emsDescriptor.filename = arg0.filename
}
if (typeof arg0.persist !== "undefined") {
emsDescriptor.persist = arg0.persist
}
if (typeof arg0.useExisting !== "undefined") {
emsDescriptor.useExisting = arg0.useExisting
}
if(arg0.doDataFill) {
emsDescriptor.doDataFill = arg0.doDataFill;
if (typeof arg0.dataFill === "object") {
emsDescriptor.dataFill = JSON.stringify(arg0.dataFill);
fillIsJSON = true;
} else {
emsDescriptor.dataFill = arg0.dataFill;
}
}
if (typeof arg0.doSetFEtags !== "undefined") {
emsDescriptor.doSetFEtags = arg0.doSetFEtags
}
if (typeof arg0.setFEtags !== "undefined") {
if (arg0.setFEtags === "full") {
emsDescriptor.setFEtagsFull = true;
} else {
emsDescriptor.setFEtagsFull = false;
}
}
if (typeof arg0.hashFunc !== "undefined") {
emsDescriptor.hashFunc = arg0.hashFunc
}
} else {
if (EMSisArray(arg0)) { // User passed in multi-dimensional array
emsDescriptor.dimensions = arg0
} else {
if (typeof arg0 === "number") { // User passed in scalar 1-D array length
emsDescriptor.dimensions = [arg0]
} else {
console.log("EMSnew: Couldn't determine type of arg0", arg0, typeof arg0)
}
}
}
if (typeof heapSize === "number") {
emsDescriptor.heapSize = heapSize;
if (heapSize <= 0 && emsDescriptor.useMap) {
console.log("Warning: New EMS array with no heap, disabling mapped keys");
emsDescriptor.useMap = false;
}
}
if (typeof filename === "string") {
emsDescriptor.filename = filename
}
}
// Compute the stride factors for each dimension of a multidimensional array
for (var dimN = 0; dimN < emsDescriptor.dimensions.length; dimN++) {
emsDescriptor.dimStride.push(emsDescriptor.nElements);
emsDescriptor.nElements *= emsDescriptor.dimensions[dimN];
}
if (typeof emsDescriptor.dimensions === "undefined") {
emsDescriptor.dimensions = [emsDescriptor.nElements];
}
// Name the region if a name wasn't given
if (!EMSisDefined(emsDescriptor.filename)) {
emsDescriptor.filename = "/EMS_region_" + this.newRegionN;
emsDescriptor.persist = false;
}
if (emsDescriptor.useExisting) {
try { fs.openSync(emsDescriptor.filename, "r"); }
catch (err) {
return;
}
}
// init() is first called from thread 0 to perform one-thread
// only operations (ie: unlinking an old file, opening a new
// file). After thread 0 has completed initialization, other
// threads can safely share the EMS array.
if (!emsDescriptor.useExisting && this.myID !== 0) EMSbarrier();
emsDescriptor.data = this.init(emsDescriptor.nElements, emsDescriptor.heapSize, // 0, 1
emsDescriptor.useMap, emsDescriptor.filename, // 2, 3
emsDescriptor.persist, emsDescriptor.useExisting, // 4, 5
emsDescriptor.doDataFill, fillIsJSON, // 6, 7
emsDescriptor.dataFill, // 8
emsDescriptor.doSetFEtags, // 9
emsDescriptor.setFEtagsFull, // 10
this.myID, this.pinThreads, this.nThreads, // 11, 12, 13
emsDescriptor.mlock); // 14
if (!emsDescriptor.useExisting && this.myID === 0) EMSbarrier();
emsDescriptor.regionN = this.newRegionN;
emsDescriptor.push = EMSpush;
emsDescriptor.pop = EMSpop;
emsDescriptor.enqueue = EMSenqueue;
emsDescriptor.dequeue = EMSdequeue;
emsDescriptor.setTag = EMSsetTag;
emsDescriptor.write = EMSwrite;
emsDescriptor.writeEF = EMSwriteEF;
emsDescriptor.writeXF = EMSwriteXF;
emsDescriptor.writeXE = EMSwriteXE;
emsDescriptor.read = EMSread;
emsDescriptor.readRW = EMSreadRW;
emsDescriptor.releaseRW = EMSreleaseRW;
emsDescriptor.readFE = EMSreadFE;
emsDescriptor.readFF = EMSreadFF;
emsDescriptor.faa = EMSfaa;
emsDescriptor.cas = EMScas;
emsDescriptor.sync = EMSsync;
emsDescriptor.index2key = EMSindex2key;
emsDescriptor.destroy = EMSdestroy;
this.newRegionN++;
EMSbarrier();
// Wrap the object with a proxy
if(emsDescriptor.ES6proxies) {
// Setter/Getter methods for the proxy object
// If the target is built into EMS use the built-in object,
// otherwise read/write the EMS value without honoring the Full/Empty tag
var EMSproxyHandler = {
get: function(target, name) {
if (name in target) {
return target[name];
} else {
return target.read(name);
}
},
set: function(target, name, value) {
target.write(name, value);
return true
}
};
try {
emsDescriptor = new Proxy(emsDescriptor, EMSproxyHandler);
}
catch (err) {
// console.log("Harmony proxies not supported:", err);
}
}
return emsDescriptor;
}
//==================================================================
// EMS object initialization, invoked by the require statement
//
function ems_wrapper(nThreadsArg, pinThreadsArg, threadingType, filename) {
var retObj = {tasks: []};
// TODO: Determining the thread ID should be done via shared memory
if (process.env.EMS_Subtask !== undefined ) {
retObj.myID = parseInt(process.env.EMS_Subtask);
} else {
retObj.myID = 0;
}
var pinThreads = false;
if (typeof pinThreadsArg === "boolean") {
pinThreads = pinThreadsArg;
}
var nThreads;
nThreads = parseInt(nThreadsArg);
if (!(nThreads > 0)) {
if (process.env.EMS_Ntasks !== undefined) {
nThreads = parseInt(process.env.EMS_Ntasks);
if (!(nThreads > 0)) {
console.log("EMS: Number of nodes must be > 0. Input:", nThreadsArg, "EnvVar:", nThreads);
process.exit(1);
}
} else {
console.log("EMS: Must declare number of nodes to use. Input:" + nThreadsArg);
process.exit(1);
}
}
var domainName = "/EMS_MainDomain";
if (filename) domainName = filename;
// All arguments are defined -- now do the EMS initialization
retObj.data = EMS.initialize(0, 0, // 0= # elements, 1=Heap Size
false, // 2 = useMap
domainName, false, false, // 3=name, 4=persist, 5=useExisting
false, false, undefined, // 6=doDataFill, 7=fillIsJSON, 8=fillValue
false, false, retObj.myID, // 9=doSetFEtags, 10=setFEtags, 11=EMS myID
pinThreads, nThreads, 99); // 12=pinThread, 13=nThreads, 14=pctMlock
var targetScript;
switch (threadingType) {
case undefined:
case "bsp":
targetScript = process.argv[1];
threadingType = "bsp";
retObj.inParallelContext = true;
break;
case "fj":
targetScript = "./EMSthreadStub";
retObj.inParallelContext = false;
break;
case "user":
targetScript = undefined;
retObj.inParallelContext = false;
break;
default:
console.log("EMS: Unknown threading model type:", threadingType);
retObj.inParallelContext = false;
break;
}
// The master thread has completed initialization, other threads may now
// safely execute.
if (targetScript !== undefined && retObj.myID === 0) {
var emsThreadStub =
"// Automatically Generated EMS Slave Thread Script\n" +
"// To edit this file, see ems.js:emsThreadStub()\n" +
"var ems = require(\"ems\")(parseInt(process.env.EMS_Ntasks));\n" +
"process.on(\"message\", function(msg) {\n" +
" eval(\"func = \" + msg.func);\n" +
" func.apply(null, msg.args);\n" +
"} );\n";
fs.writeFileSync('./EMSthreadStub.js', emsThreadStub, {flag: 'w+'});
process.env.EMS_Ntasks = nThreads;
for (var taskN = 1; taskN < nThreads; taskN++) {
process.env.EMS_Subtask = taskN;
retObj.tasks.push(
child_process.fork(targetScript,
process.argv.slice(2, process.argv.length)));
}
}
retObj.nThreads = nThreads;
retObj.threadingType = threadingType;
retObj.pinThreads = pinThreads;
retObj.domainName = domainName;
retObj.newRegionN = 0;
retObj.init = EMS.initialize;
retObj.new = EMSnew;
retObj.critical = EMScritical;
retObj.criticalEnter = EMS.criticalEnter;
retObj.criticalExit = EMS.criticalExit;
retObj.master = EMSmaster;
retObj.single = EMSsingle;
retObj.diag = EMSdiag;
retObj.parallel = EMSparallel;
retObj.barrier = EMSbarrier;
retObj.parForEach = EMSparForEach;
retObj.tmStart = EMStmStart;
retObj.tmEnd = EMStmEnd;
retObj.loopInit = EMS.loopInit;
retObj.loopChunk = EMS.loopChunk;
EMSglobal = retObj;
EMSglobal.mmapID = retObj.data.mmapID;
return retObj;
}
ems_wrapper.initialize = ems_wrapper;
module.exports = ems_wrapper;
================================================
FILE: nodejs/nodejs.cc
================================================
/*-----------------------------------------------------------------------------+
| Extended Memory Semantics (EMS) Version 1.6.1 |
| http://mogill.com/ jace@mogill.com |
+-----------------------------------------------------------------------------+
| Copyright (c) 2011-2014, Synthetic Semantics LLC. All rights reserved. |
| Copyright (c) 2019 Aleksander J Budzynowski. Update NAN to N-API. |
| Copyright (c) 2015-2020, Jace A Mogill. All rights reserved. |
| |
| Redistribution and use in source and binary forms, with or without |
| modification, are permitted provided that the following conditions are met: |
| * Redistributions of source code must retain the above copyright |
| notice, this list of conditions and the following disclaimer. |
| * Redistributions in binary form must reproduce the above copyright |
| notice, this list of conditions and the following disclaimer in the |
| documentation and/or other materials provided with the distribution. |
| * Neither the name of the Synthetic Semantics nor the names of its |
| contributors may be used to endorse or promote products derived |
| from this software without specific prior written permission. |
| |
| THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS |
| "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT |
| LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR |
| A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL SYNTHETIC |
| SEMANTICS LLC BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, |
| EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, |
| PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR |
| PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF |
| LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING |
| NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS |
| SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. |
| |
+-----------------------------------------------------------------------------*/
#include
#include "nodejs.h"
#include "../src/ems.h"
#include "../src/ems_types.h"
/**
* Convert a NAPI object to an EMS object stored on the stack
* @param napiValue Source Napi object
* @param emsValue Target EMS object
* @param stringIsJSON You know who
* @return True if successful converting
*/
#define NAPI_OBJ_2_EMS_OBJ(napiValue, emsValue, stringIsJSON) { \
emsValue.type = NapiObjToEMStype(napiValue, stringIsJSON); \
switch (emsValue.type) { \
case EMS_TYPE_BOOLEAN: { \
bool tmp = napiValue.As(); \
emsValue.value = (void *) tmp; \
break; \
} \
case EMS_TYPE_INTEGER: { \
int64_t tmp = napiValue.As(); \
emsValue.value = (void *) tmp; \
} \
break; \
case EMS_TYPE_FLOAT: { \
EMSulong_double alias = {.d = napiValue.As()};\
emsValue.value = (void *) alias.u64; \
} \
break; \
case EMS_TYPE_JSON: \
case EMS_TYPE_STRING: { \
std::string s = napiValue.As().Utf8Value(); \
emsValue.length = s.length() + 1; /* +1 for trailing NULL */\
struct rlimit rl; \
if (getrlimit(RLIMIT_STACK, &rl) == 0) { \
if (emsValue.length > rl.rlim_cur) { \
THROW_TYPE_ERROR(QUOTE(__FUNCTION__) " ERROR: Unable to allocate scratch memory for serialized value. Exceeds current stack size limit."); \
} \
} else { \
THROW_ERROR(" ERROR: Unable to get stack limit."); \
} \
emsValue.value = alloca(emsValue.length); \
memcpy(emsValue.value, s.c_str(), emsValue.length); \
} \
break; \
case EMS_TYPE_UNDEFINED: \
emsValue.value = (void *) 0xbeeff00d; \
break; \
default: \
THROW_TYPE_ERROR(QUOTE(__FUNCTION__) " ERROR: Invalid value type");\
} \
}
/**
* Convert an EMS object into a Napi object
* @param env Napi Env object
* @param emsValue Source EMS object
* @return converted value
*/
static Napi::Value inline
ems2napiReturnValue(Napi::Env env, EMSvalueType *emsValue) {
switch(emsValue->type) {
case EMS_TYPE_BOOLEAN: {
return Napi::Value::From(env, (bool) emsValue->value);
}
break;
case EMS_TYPE_INTEGER: {
int32_t retInt = ((int64_t) emsValue->value) & 0xffffffff; /* TODO: Bug -- only 32 bits of 64? */
return Napi::Number::New(env, retInt);
}
break;
case EMS_TYPE_FLOAT: {
EMSulong_double alias = {.u64 = (uint64_t) emsValue->value};
return Napi::Number::New(env, alias.d);
}
break;
case EMS_TYPE_JSON: {
Napi::Object retObj = Napi::Object::New(env);
retObj.Set("data", Napi::String::New(env, (char *) emsValue->value));
return retObj;
}
break;
case EMS_TYPE_STRING: {
return Napi::String::New(env, (char *) emsValue->value);
}
break;
case EMS_TYPE_UNDEFINED: {
return env.Undefined();
}
break;
default:
THROW_TYPE_ERROR("ems2napiReturnValue - ERROR: Invalid type of data read from memory");
}
}
Napi::Value NodeJScriticalEnter(const Napi::CallbackInfo& info) {
Napi::Env env = info.Env();
NODE_MMAPID_DECL;
int64_t timeout;
if (info.Length() == 1) {
timeout = info[0].As();
} else {
THROW_ERROR("NodeJScriticalEner: invalid or missing timeout duration");
}
int timeRemaining = EMScriticalEnter(mmapID, (int) timeout);
if (timeRemaining <= 0) {
THROW_ERROR("NodeJScriticalEnter: Unable to enter critical region before timeout");
} else {
return Napi::Value::From(env, timeRemaining);
}
}
Napi::Value NodeJScriticalExit(const Napi::CallbackInfo& info) {
Napi::Env env = info.Env();
NODE_MMAPID_DECL;
bool success = EMScriticalExit(mmapID);
if (!success) {
THROW_ERROR("NodeJScriticalExit: critical region mutex lost while locked?!");
} else {
return Napi::Value::From(env, success);
}
}
Napi::Value NodeJSbarrier(const Napi::CallbackInfo& info) {
Napi::Env env = info.Env();
NODE_MMAPID_DECL;
int timeout;
if (info.Length() == 1) {
timeout = info[0].As();
} else {
THROW_ERROR("NodeJSbarrier: invalid or missing timeout duration");
}
int timeRemaining = EMSbarrier(mmapID, timeout);
if (timeRemaining <= 0) {
THROW_ERROR("NodeJSbarrer: Failed to sync at barrier");
} else {
return Napi::Value::From(env, timeRemaining);
}
}
Napi::Value NodeJSsingleTask(const Napi::CallbackInfo& info) {
Napi::Env env = info.Env();
NODE_MMAPID_DECL;
bool did_work = EMSsingleTask(mmapID);
return Napi::Boolean::New(env, did_work);
}
Napi::Value NodeJScas(const Napi::CallbackInfo& info) {
Napi::Env env = info.Env();
EMSvalueType returnValue = EMS_VALUE_TYPE_INITIALIZER;
STACK_ALLOC_AND_CHECK_KEY_ARG;
EMSvalueType oldVal = EMS_VALUE_TYPE_INITIALIZER;
EMSvalueType newVal = EMS_VALUE_TYPE_INITIALIZER;
if (info.Length() != 3) {
THROW_ERROR(SOURCE_LOCATION ": Called with wrong number of args.");
}
NAPI_OBJ_2_EMS_OBJ(info[1], oldVal, false);
NAPI_OBJ_2_EMS_OBJ(info[2], newVal, false);
if (!EMScas(mmapID, &key, &oldVal, &newVal, &returnValue)) {
THROW_ERROR("NodeJScas: Failed to get a valid old value");
}
return ems2napiReturnValue(env, &returnValue);
}
Napi::Value NodeJSfaa(const Napi::CallbackInfo& info) {
Napi::Env env = info.Env();
EMSvalueType returnValue = EMS_VALUE_TYPE_INITIALIZER;
STACK_ALLOC_AND_CHECK_KEY_ARG;
STACK_ALLOC_AND_CHECK_VALUE_ARG(1);
bool success = EMSfaa(mmapID, &key, &value, &returnValue);
if (!success) {
THROW_ERROR("NodeJSfaa: Failed to get a valid old value");
}
return ems2napiReturnValue(env, &returnValue);
}
Napi::Value NodeJSpush(const Napi::CallbackInfo& info) {
Napi::Env env = info.Env();
NODE_MMAPID_DECL;
STACK_ALLOC_AND_CHECK_VALUE_ARG(0);
int returnValue = EMSpush(mmapID, &value);
return Napi::Value::From(env, returnValue);
}
Napi::Value NodeJSpop(const Napi::CallbackInfo& info) {
Napi::Env env = info.Env();
EMSvalueType returnValue = EMS_VALUE_TYPE_INITIALIZER;
NODE_MMAPID_DECL;
bool success = EMSpop(mmapID, &returnValue);
if (!success) {
THROW_ERROR("NodeJSpop: Failed to pop a value off the stack");
}
return ems2napiReturnValue(env, &returnValue);
}
Napi::Value NodeJSenqueue(const Napi::CallbackInfo& info) {
Napi::Env env = info.Env();
NODE_MMAPID_DECL;
STACK_ALLOC_AND_CHECK_VALUE_ARG(0);
int returnValue = EMSenqueue(mmapID, &value);
return Napi::Value::From(env, returnValue);
}
Napi::Value NodeJSdequeue(const Napi::CallbackInfo& info) {
Napi::Env env = info.Env();
EMSvalueType returnValue = EMS_VALUE_TYPE_INITIALIZER;
NODE_MMAPID_DECL;
bool success = EMSdequeue(mmapID, &returnValue);
if (!success) {
THROW_ERROR("NodeJSdequeue: Failed to dequeue a value");
}
return ems2napiReturnValue(env, &returnValue);
}
Napi::Value NodeJSloopInit(const Napi::CallbackInfo& info) {
Napi::Env env = info.Env();
NODE_MMAPID_DECL;
if (info.Length() != 4) {
THROW_ERROR("NodeJSloopInit: Wrong number of args");
}
int32_t start = (int32_t) info[0].As();
int32_t end = (int32_t) info[1].As();
int schedule_mode;
std::string sched_string = info[2].As().Utf8Value();
if (sched_string.compare("guided") == 0) {
schedule_mode = EMS_SCHED_GUIDED;
} else {
if (sched_string.compare("dynamic") == 0) {
schedule_mode = EMS_SCHED_DYNAMIC;
} else {
THROW_ERROR("NodeJSloopInit: Unknown/invalid schedule mode");
}
}
int32_t minChunk = (int32_t) info[3].As();
bool success = EMSloopInit(mmapID, start, end, minChunk, schedule_mode);
if (!success) {
THROW_ERROR("NodeJSloopInit: Unknown failure to initalize loop");
} else {
return Napi::Value::From(env, success);
}
}
Napi::Value NodeJSloopChunk(const Napi::CallbackInfo& info) {
Napi::Env env = info.Env();
NODE_MMAPID_DECL;
if (info.Length() != 0) {
THROW_ERROR("NodeJSloopChunk: Arguments provided, but none accepted");
}
int32_t start, end;
EMSloopChunk(mmapID, &start, &end); // Unusued return value
Napi::Object retObj = Napi::Object::New(env);
retObj.Set("start", Napi::Value::From(env, start));
retObj.Set("end", Napi::Value::From(env, end));
return retObj;
}
//--------------------------------------------------------------
Napi::Value NodeJSread(const Napi::CallbackInfo& info) {
Napi::Env env = info.Env();
EMSvalueType returnValue = EMS_VALUE_TYPE_INITIALIZER;
STACK_ALLOC_AND_CHECK_KEY_ARG;
if (!EMSread(mmapID, &key, &returnValue)) {
THROW_ERROR(QUOTE(__FUNCTION__) ": Unable to read (no return value) from EMS.");
} else {
return ems2napiReturnValue(env, &returnValue);
}
}
Napi::Value NodeJSreadFE(const Napi::CallbackInfo& info) {
Napi::Env env = info.Env();
EMSvalueType returnValue = EMS_VALUE_TYPE_INITIALIZER;
STACK_ALLOC_AND_CHECK_KEY_ARG;
if (!EMSreadFE(mmapID, &key, &returnValue)) {
THROW_ERROR(QUOTE(__FUNCTION__) ": Unable to read (no return value) from EMS.");
} else {
return ems2napiReturnValue(env, &returnValue);
}
}
Napi::Value NodeJSreadFF(const Napi::CallbackInfo& info) {
Napi::Env env = info.Env();
EMSvalueType returnValue = EMS_VALUE_TYPE_INITIALIZER;
STACK_ALLOC_AND_CHECK_KEY_ARG;
if (!EMSreadFF(mmapID, &key, &returnValue)) {
THROW_ERROR(QUOTE(__FUNCTION__) ": Unable to read (no return value) from EMS.");
} else {
return ems2napiReturnValue(env, &returnValue);
}
}
Napi::Value NodeJSreadRW(const Napi::CallbackInfo& info) {
Napi::Env env = info.Env();
EMSvalueType returnValue = EMS_VALUE_TYPE_INITIALIZER;
STACK_ALLOC_AND_CHECK_KEY_ARG;
if (!EMSreadRW(mmapID, &key, &returnValue)) {
THROW_ERROR(QUOTE(__FUNCTION__) ": Unable to read (no return value) from EMS.");
} else {
return ems2napiReturnValue(env, &returnValue);
}
}
Napi::Value NodeJSreleaseRW(const Napi::CallbackInfo& info) {
Napi::Env env = info.Env();
STACK_ALLOC_AND_CHECK_KEY_ARG;
int nReadersActive = EMSreleaseRW(mmapID, &key);
if (nReadersActive < 0) {
THROW_ERROR("NodeJSreleaseRW: Invalid index for key, or index key in bad state");
} else {
return Napi::Value::From(env, nReadersActive);
}
}
// ====================================================
Napi::Value NodeJSwrite(const Napi::CallbackInfo& info) {
Napi::Env env = info.Env();
STACK_ALLOC_AND_CHECK_KEY_ARG;
STACK_ALLOC_AND_CHECK_VALUE_ARG(1);
bool returnValue = EMSwrite(mmapID, &key, &value);
return Napi::Value::From(env, returnValue);
}
Napi::Value NodeJSwriteEF(const Napi::CallbackInfo& info) {
Napi::Env env = info.Env();
STACK_ALLOC_AND_CHECK_KEY_ARG;
STACK_ALLOC_AND_CHECK_VALUE_ARG(1);
bool returnValue = EMSwriteEF(mmapID, &key, &value);
return Napi::Value::From(env, returnValue);
}
Napi::Value NodeJSwriteXF(const Napi::CallbackInfo& info) {
Napi::Env env = info.Env();
STACK_ALLOC_AND_CHECK_KEY_ARG;
STACK_ALLOC_AND_CHECK_VALUE_ARG(1);
bool returnValue = EMSwriteXF(mmapID, &key, &value);
return Napi::Value::From(env, returnValue);
}
Napi::Value NodeJSwriteXE(const Napi::CallbackInfo& info) {
Napi::Env env = info.Env();
STACK_ALLOC_AND_CHECK_KEY_ARG;
STACK_ALLOC_AND_CHECK_VALUE_ARG(1);
bool returnValue = EMSwriteXE(mmapID, &key, &value);
return Napi::Value::From(env, returnValue);
}
Napi::Value NodeJSsetTag(const Napi::CallbackInfo& info) {
Napi::Env env = info.Env();
STACK_ALLOC_AND_CHECK_KEY_ARG;
STACK_ALLOC_AND_CHECK_VALUE_ARG(1); // Bool -- is full
bool success = EMSsetTag(mmapID, &key, (bool)value.value);
if (success) {
return Napi::Value::From(env, true);
} else {
THROW_ERROR("NodeJSsetTag: Invalid key, unable to set tag");
}
}
Napi::Value NodeJSsync(const Napi::CallbackInfo& info) {
Napi::Env env = info.Env();
NODE_MMAPID_DECL;
fprintf(stderr, "NodeJSsync: WARNING: sync is not implemented\n");
bool success = EMSsync(mmapID);
return Napi::Value::From(env, success);
}
Napi::Value NodeJSindex2key(const Napi::CallbackInfo& info) {
Napi::Env env = info.Env();
NODE_MMAPID_DECL;
EMSvalueType key = EMS_VALUE_TYPE_INITIALIZER;
int idx = (size_t) (int64_t) info[0].As();
if ( !EMSindex2key(mmapID, idx, &key) ) {
fprintf(stderr, "NodeJSindex2key: Error converting index (%d) to key\n", idx);
}
return ems2napiReturnValue(env, &key);
}
Napi::Value NodeJSdestroy(const Napi::CallbackInfo& info) {
Napi::Env env = info.Env();
NODE_MMAPID_DECL;
bool do_unlink = info[0].As();
bool success = EMSdestroy(mmapID, do_unlink);
if (success) {
return Napi::Value::From(env, true);
} else {
THROW_ERROR("NodeJSdestroy: Failed to destroy EMS array");
}
};
//==================================================================
// EMS Entry Point: Allocate and initialize the EMS domain memory
Napi::Value NodeJSinitialize(const Napi::CallbackInfo& info) {
Napi::Env env = info.Env();
if (info.Length() != 15) {
THROW_ERROR("NodeJSinitialize: Incorrect number of arguments");
}
EMSvalueType fillData = EMS_VALUE_TYPE_INITIALIZER;
// Parse all the arguments
int64_t nElements = info[0].As();
size_t heapSize = (int64_t) info[1].As();
bool useMap = info[2].As();
std::string filestr(info[3].As().Utf8Value());
const char *filename = filestr.c_str();
bool persist = info[4].As();
bool useExisting = info[5].As();
bool doDataFill = info[6].As();
bool fillIsJSON = info[7].As();
// 8 = Data Fill type TBD during fill
bool doSetFEtags = info[9].As();
bool setFEtags = info[10].As();
EMSmyID = (int) info[11].As();
bool pinThreads = info[12].As();
int32_t nThreads = info[13].As();
int32_t pctMLock = info[14].As();
if (doDataFill) {
NAPI_OBJ_2_EMS_OBJ(info[8], fillData, fillIsJSON);
if (doDataFill && (fillData.type == EMS_TYPE_JSON || fillData.type == EMS_TYPE_STRING)) {
// Copy the default values to the heap because napiObj2EMSval copies them to the stack
void *valueOnStack = fillData.value;
fillData.value = malloc(fillData.length + 1);
if (fillData.value == NULL) {
THROW_ERROR("NodeJSinitialize: failed to allocate the default value's storage on the heap");
}
memcpy(fillData.value, valueOnStack, fillData.length + 1);
}
}
int emsBufN = EMSinitialize(nElements, // 0
heapSize, // 1
useMap, // 2
filename, // 3
persist, // 4
useExisting, // 5
doDataFill, // 6 Data Fill type TBD during fill
fillIsJSON, // 7
&fillData, // 8
doSetFEtags, // 9
setFEtags, // 10
EMSmyID, // 11
pinThreads, // 12
nThreads, // 13
pctMLock); // 14
if (emsBufN < 0) {
THROW_ERROR("NodeJSinitialize: failed to initialize EMS array");
}
// ========================================================================================
Napi::Object obj = Napi::Object::New(env);
obj.Set(Napi::String::New(env, "mmapID"), Napi::Value::From(env, emsBufN));
ADD_FUNC_TO_NAPI_OBJ(obj, "faa", NodeJSfaa);
ADD_FUNC_TO_NAPI_OBJ(obj, "cas", NodeJScas);
ADD_FUNC_TO_NAPI_OBJ(obj, "read", NodeJSread);
ADD_FUNC_TO_NAPI_OBJ(obj, "write", NodeJSwrite);
ADD_FUNC_TO_NAPI_OBJ(obj, "readRW", NodeJSreadRW);
ADD_FUNC_TO_NAPI_OBJ(obj, "releaseRW", NodeJSreleaseRW);
ADD_FUNC_TO_NAPI_OBJ(obj, "readFE", NodeJSreadFE);
ADD_FUNC_TO_NAPI_OBJ(obj, "readFF", NodeJSreadFF);
ADD_FUNC_TO_NAPI_OBJ(obj, "setTag", NodeJSsetTag);
ADD_FUNC_TO_NAPI_OBJ(obj, "writeEF", NodeJSwriteEF);
ADD_FUNC_TO_NAPI_OBJ(obj, "writeXF", NodeJSwriteXF);
ADD_FUNC_TO_NAPI_OBJ(obj, "writeXE", NodeJSwriteXE);
ADD_FUNC_TO_NAPI_OBJ(obj, "push", NodeJSpush);
ADD_FUNC_TO_NAPI_OBJ(obj, "pop", NodeJSpop);
ADD_FUNC_TO_NAPI_OBJ(obj, "enqueue", NodeJSenqueue);
ADD_FUNC_TO_NAPI_OBJ(obj, "dequeue", NodeJSdequeue);
ADD_FUNC_TO_NAPI_OBJ(obj, "sync", NodeJSsync);
ADD_FUNC_TO_NAPI_OBJ(obj, "index2key", NodeJSindex2key);
ADD_FUNC_TO_NAPI_OBJ(obj, "destroy", NodeJSdestroy);
return obj;
}
//---------------------------------------------------------------
static Napi::Object RegisterModule(Napi::Env env, Napi::Object exports) {
ADD_FUNC_TO_NAPI_OBJ(exports, "initialize", NodeJSinitialize);
ADD_FUNC_TO_NAPI_OBJ(exports, "barrier", NodeJSbarrier);
ADD_FUNC_TO_NAPI_OBJ(exports, "singleTask", NodeJSsingleTask);
ADD_FUNC_TO_NAPI_OBJ(exports, "criticalEnter", NodeJScriticalEnter);
ADD_FUNC_TO_NAPI_OBJ(exports, "criticalExit", NodeJScriticalExit);
ADD_FUNC_TO_NAPI_OBJ(exports, "loopInit", NodeJSloopInit);
ADD_FUNC_TO_NAPI_OBJ(exports, "loopChunk", NodeJSloopChunk);
return exports;
}
NODE_API_MODULE(ems, RegisterModule);
================================================
FILE: nodejs/nodejs.h
================================================
/*-----------------------------------------------------------------------------+
| Extended Memory Semantics (EMS) Version 1.6.1 |
| http://mogill.com/ jace@mogill.com |
+-----------------------------------------------------------------------------+
| Copyright (c) 2011-2014, Synthetic Semantics LLC. All rights reserved. |
| Copyright (c) 2019 Aleksander J Budzynowski. Update NAN to N-API. |
| Copyright (c) 2015-2020, Jace A Mogill. All rights reserved. |
| |
| Redistribution and use in source and binary forms, with or without |
| modification, are permitted provided that the following conditions are met: |
| * Redistributions of source code must retain the above copyright |
| notice, this list of conditions and the following disclaimer. |
| * Redistributions in binary form must reproduce the above copyright |
| notice, this list of conditions and the following disclaimer in the |
| documentation and/or other materials provided with the distribution. |
| * Neither the name of the Synthetic Semantics nor the names of its |
| contributors may be used to endorse or promote products derived |
| from this software without specific prior written permission. |
| |
| THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS |
| "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT |
| LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR |
| A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL SYNTHETIC |
| SEMANTICS LLC BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, |
| EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, |
| PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR |
| PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF |
| LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING |
| NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS |
| SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. |
| |
+-----------------------------------------------------------------------------*/
#ifndef EMSPROJ_NODEJS_H
#define EMSPROJ_NODEJS_H
#define __STDC_LIMIT_MACROS
#include
#include "napi.h"
#define QUOTE_ARG(x) #x
#define QUOTE(x) QUOTE_ARG(x)
#define THROW_ERROR(error) { \
Napi::Error::New(env, error) \
.ThrowAsJavaScriptException(); \
return env.Null(); \
}
#define THROW_TYPE_ERROR(error) { \
Napi::TypeError::New(env, error) \
.ThrowAsJavaScriptException(); \
return env.Null(); \
}
#define ADD_FUNC_TO_NAPI_OBJ(obj, func_name, func) \
{ \
Napi::Function fn = Napi::Function::New(env, func, func_name); \
obj.Set(Napi::Value::From(env, func_name), fn); \
}
#define IS_INTEGER(x) ((double)(int64_t)(x) == (double)x)
//==================================================================
// Determine the EMS type of a Napi argument
#define NapiObjToEMStype(arg, stringIsJSON) \
( \
arg.IsNumber() ? \
(IS_INTEGER(arg.As()) ? EMS_TYPE_INTEGER : \
EMS_TYPE_FLOAT ) : \
(arg.IsString() && !stringIsJSON) ? EMS_TYPE_STRING : \
(arg.IsString() && stringIsJSON) ? EMS_TYPE_JSON : \
arg.IsBoolean() ? EMS_TYPE_BOOLEAN : \
arg.IsUndefined() ? EMS_TYPE_UNDEFINED: \
EMS_TYPE_INVALID \
)
#define SOURCE_LOCATION __FILE__ ":" QUOTE(__LINE__)
#define STACK_ALLOC_AND_CHECK_KEY_ARG \
NODE_MMAPID_DECL; \
EMSvalueType key = EMS_VALUE_TYPE_INITIALIZER; \
if (info.Length() < 1) { \
Napi::Error::New(env, SOURCE_LOCATION ": missing key argument.").ThrowAsJavaScriptException(); \
} else { \
NAPI_OBJ_2_EMS_OBJ(info[0], key, false); \
}
#define STACK_ALLOC_AND_CHECK_VALUE_ARG(argNum) \
bool stringIsJSON = false; \
EMSvalueType value = EMS_VALUE_TYPE_INITIALIZER; \
if (info.Length() == argNum + 2) { \
stringIsJSON = info[argNum + 1].As(); \
} \
if (info.Length() < argNum + 1) { \
Napi::Error::New(env, SOURCE_LOCATION ": ERROR, wrong number of arguments for value").ThrowAsJavaScriptException(); \
} else { \
NAPI_OBJ_2_EMS_OBJ(info[argNum], value, stringIsJSON); \
}
#define NODE_MMAPID_DECL \
const int mmapID = (int) info.This().As() \
.Get("mmapID").As()
Napi::Value NodeJScriticalEnter(const Napi::CallbackInfo& info);
Napi::Value NodeJScriticalExit(const Napi::CallbackInfo& info);
Napi::Value NodeJSbarrier(const Napi::CallbackInfo& info);
Napi::Value NodeJSsingleTask(const Napi::CallbackInfo& info);
Napi::Value NodeJScas(const Napi::CallbackInfo& info);
Napi::Value NodeJSfaa(const Napi::CallbackInfo& info);
Napi::Value NodeJSpush(const Napi::CallbackInfo& info);
Napi::Value NodeJSpop(const Napi::CallbackInfo& info);
Napi::Value NodeJSenqueue(const Napi::CallbackInfo& info);
Napi::Value NodeJSdequeue(const Napi::CallbackInfo& info);
Napi::Value NodeJSloopInit(const Napi::CallbackInfo& info);
Napi::Value NodeJSloopChunk(const Napi::CallbackInfo& info);
//--------------------------------------------------------------
Napi::Value NodeJSread(const Napi::CallbackInfo& info);
Napi::Value NodeJSreadRW(const Napi::CallbackInfo& info);
Napi::Value NodeJSreleaseRW(const Napi::CallbackInfo& info);
Napi::Value NodeJSreadFE(const Napi::CallbackInfo& info);
Napi::Value NodeJSreadFF(const Napi::CallbackInfo& info);
Napi::Value NodeJSwrite(const Napi::CallbackInfo& info);
Napi::Value NodeJSwriteEF(const Napi::CallbackInfo& info);
Napi::Value NodeJSwriteXF(const Napi::CallbackInfo& info);
Napi::Value NodeJSwriteXE(const Napi::CallbackInfo& info);
Napi::Value NodeJSsetTag(const Napi::CallbackInfo& info);
Napi::Value NodeJSsync(const Napi::CallbackInfo& info);
Napi::Value NodeJSindex2key(const Napi::CallbackInfo& info);
Napi::Value NodeJSdestroy(const Napi::CallbackInfo& info);
#endif //EMSPROJ__H
================================================
FILE: package.json
================================================
{
"name": "ems",
"version": "1.6.1",
"author": "Jace A Mogill ",
"description": "Persistent Shared Memory and Parallel Programming Model",
"contributors": [
{
"name": "Jace A Mogill",
"email": "mogill@mogill.com"
},
{
"name": "Aleksander Budzynowski",
"email": "aleks@budzynowski.com"
}
],
"scripts": {
"test": "cd Tests && rm -f EMSthreadStub.js && for test in `ls *js`; do node $test 8; err=$?; echo $test \": ERROR=\" $err; if [ $err -ne 0 ] ; then exit 1; fi; done",
"example": "cd Examples; node concurrent_Q_and_TM.js 8"
},
"main": "nodejs/ems.js",
"repository": {
"type": "git",
"url": "https://github.com/mogill/ems.git"
},
"homepage": "https://github.com/mogill/ems.git",
"keywords": [
"non volatile memory",
"NVM",
"NVMe",
"multithreading",
"multithreaded",
"parallel",
"parallelism",
"concurrency",
"shared memory",
"multicore",
"manycore",
"transactional memory",
"TM",
"persistent memory",
"pmem",
"Extended Memory Semantics",
"EMS"
],
"license": "BSD-3-Clause",
"engines": {
"node": ">=4.0"
},
"dependencies": {
"bindings": "^1.3.0",
"node-addon-api": "*"
}
}
================================================
FILE: src/collectives.cc
================================================
/*-----------------------------------------------------------------------------+
| Extended Memory Semantics (EMS) Version 1.3.0 |
| Synthetic Semantics http://www.synsem.com/ mogill@synsem.com |
+-----------------------------------------------------------------------------+
| Copyright (c) 2011-2014, Synthetic Semantics LLC. All rights reserved. |
| Copyright (c) 2015-2016, Jace A Mogill. All rights reserved. |
| |
| Redistribution and use in source and binary forms, with or without |
| modification, are permitted provided that the following conditions are met: |
| * Redistributions of source code must retain the above copyright |
| notice, this list of conditions and the following disclaimer. |
| * Redistributions in binary form must reproduce the above copyright |
| notice, this list of conditions and the following disclaimer in the |
| documentation and/or other materials provided with the distribution. |
| * Neither the name of the Synthetic Semantics nor the names of its |
| contributors may be used to endorse or promote products derived |
| from this software without specific prior written permission. |
| |
| THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS |
| "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT |
| LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR |
| A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL SYNTHETIC |
| SEMANTICS LLC BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, |
| EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, |
| PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR |
| PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF |
| LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING |
| NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS |
| SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. |
| |
+-----------------------------------------------------------------------------*/
#include "ems.h"
//==================================================================
// Execute Once -- Single execution
// OpenMP style, only first thread executes, remaining skip
bool EMSsingleTask(int mmapID) {
void *emsBuf = emsBufs[mmapID];
int32_t *bufInt32 = (int32_t *) emsBuf;
// Increment the tally of threads that have reached this statement
int retval = __sync_fetch_and_add(&bufInt32[EMS_CB_SINGLE], 1);
// If all threads have passed through the counter, reset fo next time
if (retval == bufInt32[EMS_CB_NTHREADS] - 1) {
bufInt32[EMS_CB_SINGLE] = 0;
}
// Return True if this thread was first to the counter, later
// threads return false
return retval == 0;
}
//==================================================================
// Critical Region Entry -- 1 thread at a time passes this barrier
//
int EMScriticalEnter(int mmapID, int timeout) {
RESET_NAP_TIME;
void *emsBuf = emsBufs[mmapID];
int32_t *bufInt32 = (int32_t *) emsBuf;
// Acquire the mutual exclusion lock
while (!__sync_bool_compare_and_swap(&(bufInt32[EMS_CB_CRITICAL]), EMS_TAG_FULL, EMS_TAG_EMPTY)
&& timeout > 0 ) {
NANOSLEEP;
timeout -= 1;
}
return timeout;
}
//==================================================================
// Critical Region Exit
bool EMScriticalExit(int mmapID) {
void *emsBuf = emsBufs[mmapID];
int32_t *bufInt32 = (int32_t *) emsBuf;
// Test the mutual exclusion lock wasn't somehow lost
if (bufInt32[EMS_CB_CRITICAL] != EMS_TAG_EMPTY) {
return false;
}
bufInt32[EMS_CB_CRITICAL] = EMS_TAG_FULL;
return true;
}
//==================================================================
// Phase Based Global Thread Barrier
int EMSbarrier(int mmapID, int timeout) {
void *emsBuf = emsBufs[mmapID];
int32_t *bufInt32 = (int32_t *) emsBuf;
int barPhase = bufInt32[EMS_CB_BARPHASE]; // Determine current phase of barrier
int retval = __sync_fetch_and_add(&bufInt32[EMS_CB_NBAR0 + barPhase], -1);
if (retval < 0) {
fprintf(stderr, "EMSbarrier: Race condition at barrier\n");
return false;
}
if (retval == 1) {
// This thread was the last to reach the barrier,
// Reset the barrier count for this phase and graduate to the next phase
bufInt32[EMS_CB_NBAR0 + barPhase] = bufInt32[EMS_CB_NTHREADS];
bufInt32[EMS_CB_BARPHASE] = !barPhase;
} else {
// Wait for the barrier phase to change, indicating the last thread arrived
RESET_NAP_TIME;
while (timeout > 0 && barPhase == bufInt32[EMS_CB_BARPHASE]) {
NANOSLEEP;
timeout -= 1;
}
}
return timeout;
}
================================================
FILE: src/ems.cc
================================================
/*-----------------------------------------------------------------------------+
| Extended Memory Semantics (EMS) Version 1.6.1 |
| http://mogill.com/ jace@mogill.com |
+-----------------------------------------------------------------------------+
| Copyright (c) 2011-2014, Synthetic Semantics LLC. All rights reserved. |
| Copyright (c) 2019 Aleksander J Budzynowski. Update NAN to N-API. |
| Copyright (c) 2015-2020, Jace A Mogill. All rights reserved. |
| |
| Redistribution and use in source and binary forms, with or without |
| modification, are permitted provided that the following conditions are met: |
| * Redistributions of source code must retain the above copyright |
| notice, this list of conditions and the following disclaimer. |
| * Redistributions in binary form must reproduce the above copyright |
| notice, this list of conditions and the following disclaimer in the |
| documentation and/or other materials provided with the distribution. |
| * Neither the name of the Synthetic Semantics nor the names of its |
| contributors may be used to endorse or promote products derived |
| from this software without specific prior written permission. |
| |
| THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS |
| "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT |
| LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR |
| A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL SYNTHETIC |
| SEMANTICS LLC BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, |
| EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, |
| PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR |
| PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF |
| LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING |
| NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS |
| SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. |
| |
+-----------------------------------------------------------------------------*/
#include "ems.h"
//==================================================================
// Resolve External Declarations
//
int EMSmyID = -1; // EMS Process ID
char *emsBufs[EMS_MAX_N_BUFS] = { NULL };
size_t emsBufLengths[EMS_MAX_N_BUFS] = { 0 };
char emsBufFilenames[EMS_MAX_N_BUFS][MAX_FNAME_LEN] = { { 0 } };
//==================================================================
// Wrappers around memory allocator to ensure mutual exclusion
// The buddy memory allocator is not thread safe, so this is necessary for now.
// It can be used as the hook to wrap the non-threaded allocator with a
// multiplexor of several independent regions.
//
// Returns the byte offset in the EMS data space of the space allocated
//
size_t emsMutexMem_alloc(struct emsMem *heap, // Base of EMS malloc structs
size_t len, // Number of bytes to allocate
volatile char *mutex) // Pointer to the mem allocator's mutex
{
RESET_NAP_TIME;
// Wait until we acquire the allocator's mutex
while (!__sync_bool_compare_and_swap(mutex, EMS_TAG_EMPTY, EMS_TAG_FULL)) {
NANOSLEEP;
}
size_t retval = emsMem_alloc(heap, len);
*mutex = EMS_TAG_EMPTY; // Unlock the allocator's mutex
return (retval);
}
void emsMutexMem_free(struct emsMem *heap, // Base of EMS malloc structs
size_t addr, // Offset of alloc'd block in EMS memory
volatile char *mutex) // Pointer to the mem allocator's mutex
{
RESET_NAP_TIME;
// Wait until we acquire the allocator's mutex
while (!__sync_bool_compare_and_swap(mutex, EMS_TAG_EMPTY, EMS_TAG_FULL)) {
NANOSLEEP;
}
emsMem_free(heap, addr);
*mutex = EMS_TAG_EMPTY; // Unlock the allocator's mutex
}
//==================================================================
// Convert any type of key to an index
//
int64_t EMSkey2index(void *emsBuf, EMSvalueType *key, bool is_mapped) {
volatile EMStag_t *bufTags = (EMStag_t *) emsBuf;
volatile int64_t *bufInt64 = (int64_t *) emsBuf;
volatile double *bufDouble = (double *) emsBuf;
const char *bufChar = (char *) emsBuf;
int64_t idx = 0;
switch (key->type) {
case EMS_TYPE_BOOLEAN:
if ((bool) key->value) idx = 1;
else idx = 0;
break;
case EMS_TYPE_INTEGER:
idx = llabs((int64_t) key->value);
break;
case EMS_TYPE_FLOAT:
idx = llabs((int64_t) key->value);
break;
case EMS_TYPE_STRING:
idx = EMShashString((char *) key->value);
break;
case EMS_TYPE_UNDEFINED:
fprintf(stderr, "EMS ERROR: EMSkey2index keyType is defined as Undefined\n");
return -1;
default:
fprintf(stderr, "EMS ERROR: EMSkey2index keyType(%d) is unknown\n", key->type);
return -1;
}
int nTries = 0;
bool matched = false;
bool notPresent = false;
EMStag_t mapTags;
if (is_mapped) {
while (nTries < MAX_OPEN_HASH_STEPS && !matched && !notPresent) {
idx = idx % bufInt64[EMScbData(EMS_ARR_NELEM)];
// Wait until the map key is FULL, mark it busy while map lookup is performed
mapTags.byte = EMStransitionFEtag(&bufTags[EMSmapTag(idx)], NULL, EMS_TAG_FULL, EMS_TAG_BUSY, EMS_TAG_ANY);
if (mapTags.tags.type == key->type) {
switch (key->type) {
case EMS_TYPE_BOOLEAN:
case EMS_TYPE_INTEGER:
if ((int64_t) key->value == bufInt64[EMSmapData(idx)]) {
matched = true;
}
break;
case EMS_TYPE_FLOAT: {
EMSulong_double alias;
alias.u64 = (uint64_t) key->value;
if (alias.d == bufDouble[EMSmapData(idx)]) {
matched = true;
}
}
break;
case EMS_TYPE_STRING: {
int64_t keyStrOffset = bufInt64[EMSmapData(idx)];
if (strcmp((const char *) key->value, EMSheapPtr(keyStrOffset)) == 0) {
matched = true;
}
}
break;
case EMS_TYPE_UNDEFINED:
// Nothing hashed to this map index yet, so the key does not exist
notPresent = true;
break;
default:
fprintf(stderr, "EMS ERROR: EMSreadIndexMap: Unknown mem type\n");
matched = true;
}
}
if (mapTags.tags.type == EMS_TYPE_UNDEFINED) notPresent = true;
bufTags[EMSmapTag(idx)].tags.fe = EMS_TAG_FULL;
if (!matched) {
// No match, set this Map entry back to full and try again
nTries++;
idx++;
}
}
if (!matched) { idx = -1; }
}
int64_t retval = idx % bufInt64[EMScbData(EMS_ARR_NELEM)];
return retval;
}
#if 0
//==================================================================
// Callback for destruction of an EMS array
//
static void EMSarrFinalize(char *data, void *hint) {
//fprintf(stderr, "%d: EMSarrFinalize data=%lx hint=%lx %" PRIu64 "\n", EMSmyID, data, hint, hint);
munmap(data, (size_t) hint);
{
v8::HandleScope scope;
EMS_DECL(args);
size_t length = buffer->handle_->GetIndexedPropertiesExternalArrayDataLength();
if(-1 == munmap(emsBuf, length)) return v8::False();
// JQM TODO shoud unlink here?
fprintf(stderr, "Unmap -- should also unlink\n");
buffer->handle_->SetIndexedPropertiesToExternalArrayData(NULL, v8::kExternalUnsignedByteArray, 0);
buffer->handle_.Set(length_symbol, v8::Integer::NewFromUnsigned(0));
buffer->handle_.Dispose();
args.This().Set(length_symbol, v8::Integer::NewFromUnsigned(0));
return v8::True();
}
}
#endif
//==================================================================
// Wait until the FE tag is a particular state, then transition it to the new state
// Return new tag state
//
unsigned char EMStransitionFEtag(EMStag_t volatile *tag, EMStag_t volatile *mapTag,
unsigned char oldFE, unsigned char newFE, unsigned char oldType) {
RESET_NAP_TIME;
EMStag_t oldTag; // Desired tag value to start of the transition
EMStag_t newTag; // Tag value at the end of the transition
EMStag_t volatile memTag; // Tag value actually stored in memory
memTag.byte = tag->byte;
while (oldType == EMS_TAG_ANY || memTag.tags.type == oldType) {
oldTag.byte = memTag.byte; // Copy current type and RW count information
oldTag.tags.fe = oldFE; // Set the desired start tag state
newTag.byte = memTag.byte; // Copy current type and RW count information
newTag.tags.fe = newFE; // Set the final tag state
// Attempt to transition the state from old to new
memTag.byte = __sync_val_compare_and_swap(&(tag->byte), oldTag.byte, newTag.byte);
if (memTag.byte == oldTag.byte) {
return (newTag.byte);
} else {
// Allow preemptive map acquisition while waiting for data
if (mapTag) { mapTag->tags.fe = EMS_TAG_FULL; }
NANOSLEEP;
if (mapTag) { EMStransitionFEtag(mapTag, NULL, EMS_TAG_FULL, EMS_TAG_BUSY, EMS_TAG_ANY); }
memTag.byte = tag->byte; // Re-load tag in case was transitioned by another thread
}
}
return (memTag.byte);
}
//==================================================================
// Hash a string into an integer
//
int64_t EMShashString(const char *key) {
// TODO BUG MAGIC Max key length
int charN = 0;
uint64_t hash = 0;
while (key[charN] != 0) {
hash = key[charN] + (hash << 6) + (hash << 16) - hash;
charN++;
}
hash *= 1191613; // Further scramble to prevent close strings from having close indexes
return (llabs((int64_t) hash));
}
//==================================================================
// Find the matching map key, if not present, find the
// next available open address.
// Reads map key when full and marks Busy to perform comparisons,
// if it is not a match the data is marked full again, but if it does
// match, the map key is left empty and this function
// returns the index of an existing or available array element.
//
int64_t EMSwriteIndexMap(const int mmapID, EMSvalueType *key) {
char *emsBuf = emsBufs[mmapID];
volatile int64_t *bufInt64 = (int64_t *) emsBuf;
volatile char *bufChar = emsBuf;
volatile EMStag_t *bufTags = (EMStag_t *) emsBuf;
volatile double *bufDouble = (double *) emsBuf;
EMStag_t mapTags;
// If the key already exists, use it
int64_t idx = EMSkey2index(emsBuf, key, EMSisMapped);
if(idx > 0) {
// fprintf(stderr, "write index map -- key already existed\n");
return idx;
}
idx = EMSkey2index(emsBuf, key, false);
int nTries = 0;
if (EMSisMapped) {
int matched = false;
while (nTries < MAX_OPEN_HASH_STEPS && !matched) {
idx = idx % bufInt64[EMScbData(EMS_ARR_NELEM)];
// Wait until the map key is FULL, mark it busy while map lookup is performed
mapTags.byte = EMStransitionFEtag(&bufTags[EMSmapTag(idx)], NULL, EMS_TAG_FULL, EMS_TAG_BUSY, EMS_TAG_ANY);
mapTags.tags.fe = EMS_TAG_FULL; // When written back, mark FULL
if (mapTags.tags.type == key->type || mapTags.tags.type == EMS_TYPE_UNDEFINED) {
switch (mapTags.tags.type) {
case EMS_TYPE_BOOLEAN:
if ((int64_t) key->value == (bufInt64[EMSmapData(idx)] != 0)) {
matched = true;
bufTags[EMSmapTag(idx)].tags.fe = EMS_TAG_FULL;
}
break;
case EMS_TYPE_INTEGER:
if ((int64_t) key->value == bufInt64[EMSmapData(idx)]) {
matched = true;
bufTags[EMSmapTag(idx)].tags.fe = EMS_TAG_FULL;
}
break;
case EMS_TYPE_FLOAT: {
EMSulong_double alias;
alias.u64 = (uint64_t) key->value;
if (alias.d == bufDouble[EMSmapData(idx)]) {
matched = true;
bufTags[EMSmapTag(idx)].tags.fe = EMS_TAG_FULL;
}
}
break;
case EMS_TYPE_STRING: {
int64_t keyStrOffset = bufInt64[EMSmapData(idx)];
if (strcmp((const char *) key->value, (const char *) EMSheapPtr(keyStrOffset)) == 0) {
matched = true;
bufTags[EMSmapTag(idx)].tags.fe = EMS_TAG_FULL;
}
}
break;
case EMS_TYPE_UNDEFINED:
// This map key index is still unused, so there was no match and a new
// mapped element must be allocated to perform the tag bit transitions upon
bufTags[EMSmapTag(idx)].tags.type = key->type;
switch (key->type) {
case EMS_TYPE_BOOLEAN:
bufInt64[EMSmapData(idx)] = ((int64_t) key->value != 0);
break;
case EMS_TYPE_INTEGER:
bufInt64[EMSmapData(idx)] = (int64_t) key->value;
break;
case EMS_TYPE_FLOAT: {
EMSulong_double alias;
alias.u64 = (uint64_t) key->value;
bufDouble[EMSmapData(idx)] = alias.d;
}
break;
case EMS_TYPE_STRING: {
int64_t textOffset;
EMS_ALLOC(textOffset, key->length + 1, bufChar,
"EMSwriteIndexMap(string): out of memory to store string", -1);
bufInt64[EMSmapData(idx)] = textOffset;
strcpy((char *) EMSheapPtr(textOffset), (const char *) key->value);
}
break;
case EMS_TYPE_UNDEFINED:
bufInt64[EMSmapData(idx)] = 0xdeadbeef;
break;
default:
fprintf(stderr, "EMS ERROR: EMSwriteIndexMap: unknown arg type\n");
}
matched = true;
break;
default:
fprintf(stderr, "EMS ERROR: EMSwriteIndexMap: Unknown tag type (%d) on map key\n", mapTags.tags.type);
matched = true;
}
}
if (!matched) {
// No match so set this key map back to full and try the next entry
bufTags[EMSmapTag(idx)].tags.fe = EMS_TAG_FULL;
nTries++;
idx++;
}
}
} else { // Wasn't mapped, do bounds check
if (idx < 0 || idx >= bufInt64[EMScbData(EMS_ARR_NELEM)]) {
fprintf(stderr, "Wasn't mapped do bounds check\n");
idx = -1;
}
}
if (nTries >= MAX_OPEN_HASH_STEPS) {
idx = -1;
fprintf(stderr, "EMSwriteIndexMap ran out of key mappings (ntries=%d)\n", nTries);
}
return idx;
}
//==================================================================
// Read EMS memory, enforcing Full/Empty tag transitions
bool EMSreadUsingTags(const int mmapID,
EMSvalueType *key, // Index to read from
EMSvalueType *returnValue,
unsigned char initialFE, // Block until F/E tags are this value
unsigned char finalFE) // Set the tag to this value when done
{
RESET_NAP_TIME;
void *emsBuf = emsBufs[mmapID];
volatile EMStag_t *bufTags = (EMStag_t *) emsBuf;
volatile int64_t *bufInt64 = (int64_t *) emsBuf;
volatile double *bufDouble = (double *) emsBuf;
const char *bufChar = (const char *) emsBuf;
returnValue->type = EMS_TYPE_UNDEFINED;
returnValue->value = (void *) 0xdeafbeef; // TODO: Should return default value even when not doing write allocate
EMStag_t newTag, oldTag, memTag;
int64_t idx = EMSkey2index(emsBuf, key, EMSisMapped);
// Allocate on Write, writes include modification of the tag:
// If the EMS object being read is undefined and we're changing the f/e state
// then allocate the undefined object and set the state. If the state is
// not changing, do not allocate the undefined element.
if(EMSisMapped && idx < 0) {
if (finalFE != EMS_TAG_ANY) {
idx = EMSwriteIndexMap(mmapID, key);
if (idx < 0) {
fprintf(stderr, "EMSreadUsingTags: Unable to allocate on read for new map index\n");
return false;
}
} else {
return true;
}
}
if (idx < 0 || idx >= bufInt64[EMScbData(EMS_ARR_NELEM)]) {
fprintf(stderr, "EMSreadUsingTags: index out of bounds\n");
return false;
}
while (true) {
memTag.byte = bufTags[EMSdataTag(idx)].byte;
// Wait until FE tag is not FULL
if (initialFE == EMS_TAG_ANY ||
(initialFE != EMS_TAG_RW_LOCK && memTag.tags.fe == initialFE) ||
(initialFE == EMS_TAG_RW_LOCK &&
((memTag.tags.fe == EMS_TAG_RW_LOCK && newTag.tags.rw < EMS_RW_NREADERS_MAX) ||
memTag.tags.fe == EMS_TAG_FULL) &&
(memTag.tags.rw < ((1 << EMS_TYPE_NBITS_RW) - 1)) // Counter is already saturated
)
) {
newTag.byte = memTag.byte;
oldTag.byte = memTag.byte;
newTag.tags.fe = EMS_TAG_BUSY;
if (initialFE == EMS_TAG_RW_LOCK) {
newTag.tags.rw++;
} else {
oldTag.tags.fe = initialFE;
}
// Transition FE from FULL to BUSY
if (initialFE == EMS_TAG_ANY ||
__sync_bool_compare_and_swap(&(bufTags[EMSdataTag(idx)].byte), oldTag.byte, newTag.byte)) {
// Under BUSY lock:
// Read the data, then reset the FE tag, then return the original value in memory
newTag.tags.fe = finalFE;
returnValue->type = newTag.tags.type;
switch (newTag.tags.type) {
case EMS_TYPE_BOOLEAN: {
returnValue->value = (void *) (bufInt64[EMSdataData(idx)] != 0);
if (finalFE != EMS_TAG_ANY) bufTags[EMSdataTag(idx)].byte = newTag.byte;
if (EMSisMapped) bufTags[EMSmapTag(idx)].tags.fe = EMS_TAG_FULL;
return true;
}
case EMS_TYPE_INTEGER: {
returnValue->value = (void *) bufInt64[EMSdataData(idx)];
if (finalFE != EMS_TAG_ANY) bufTags[EMSdataTag(idx)].byte = newTag.byte;
if (EMSisMapped) bufTags[EMSmapTag(idx)].tags.fe = EMS_TAG_FULL;
return true;
}
case EMS_TYPE_FLOAT: {
EMSulong_double alias;
alias.d = bufDouble[EMSdataData(idx)];
returnValue->value = (void *) alias.u64;
if (finalFE != EMS_TAG_ANY) bufTags[EMSdataTag(idx)].byte = newTag.byte;
if (EMSisMapped) bufTags[EMSmapTag(idx)].tags.fe = EMS_TAG_FULL;
return true;
}
case EMS_TYPE_JSON:
case EMS_TYPE_STRING: {
returnValue->value = (void *) EMSheapPtr(bufInt64[EMSdataData(idx)]);
returnValue->length = strlen((const char *)returnValue->value);
if (finalFE != EMS_TAG_ANY) bufTags[EMSdataTag(idx)].byte = newTag.byte;
if (EMSisMapped) bufTags[EMSmapTag(idx)].tags.fe = EMS_TAG_FULL;
return true;
}
case EMS_TYPE_UNDEFINED: {
returnValue->value = (void *) 0xcafebeef;
if (finalFE != EMS_TAG_ANY) bufTags[EMSdataTag(idx)].byte = newTag.byte;
if (EMSisMapped) bufTags[EMSmapTag(idx)].tags.fe = EMS_TAG_FULL;
return true;
}
default:
fprintf(stderr, "EMSreadUsingTags: unknown type (%d) read from memory\n", newTag.tags.type);
return false;
}
} else {
// Tag was marked BUSY between test read and CAS, must retry
}
} else {
// Tag was already marked BUSY, must retry
}
// CAS failed or memory wasn't in initial state, wait and retry.
// Permit preemptive map acquisition while waiting for data.
if (EMSisMapped) { bufTags[EMSmapTag(idx)].tags.fe = EMS_TAG_FULL; }
NANOSLEEP;
if (EMSisMapped) {
EMStransitionFEtag(&bufTags[EMSmapTag(idx)], NULL, EMS_TAG_FULL, EMS_TAG_BUSY, EMS_TAG_ANY);
}
}
}
//==================================================================
// Read under multiple readers-single writer lock
bool EMSreadRW(const int mmapID, EMSvalueType *key, EMSvalueType *returnValue) {
return EMSreadUsingTags(mmapID, key, returnValue, EMS_TAG_RW_LOCK, EMS_TAG_RW_LOCK);
}
//==================================================================
// Read when full and leave empty
bool EMSreadFE(const int mmapID, EMSvalueType *key, EMSvalueType *returnValue) {
return EMSreadUsingTags(mmapID, key, returnValue, EMS_TAG_FULL, EMS_TAG_EMPTY);
}
//==================================================================
// Read when full and leave Full
bool EMSreadFF(const int mmapID, EMSvalueType *key, EMSvalueType *returnValue) {
return EMSreadUsingTags(mmapID, key, returnValue, EMS_TAG_FULL, EMS_TAG_FULL);
}
//==================================================================
// Wrapper around read
bool EMSread(const int mmapID, EMSvalueType *key, EMSvalueType *returnValue) {
return EMSreadUsingTags(mmapID, key, returnValue, EMS_TAG_ANY, EMS_TAG_ANY);
}
//==================================================================
// Decrement the reference count of the multiple readers-single writer lock
int EMSreleaseRW(const int mmapID, EMSvalueType *key) {
RESET_NAP_TIME;
void *emsBuf = emsBufs[mmapID];
volatile int64_t *bufInt64 = (int64_t *) emsBuf;
volatile EMStag_t *bufTags = (EMStag_t *) emsBuf;
EMStag_t newTag, oldTag;
int64_t idx = EMSkey2index(emsBuf, key, EMSisMapped);
if (idx < 0 || idx >= bufInt64[EMScbData(EMS_ARR_NELEM)]) {
fprintf(stderr, "EMSreleaseRW: invalid index (%" PRIi64 ")\n", idx);
return -1;
}
while (true) {
oldTag.byte = bufTags[EMSdataTag(idx)].byte;
newTag.byte = oldTag.byte;
if (oldTag.tags.fe == EMS_TAG_RW_LOCK) {
// Already under a RW lock
if (oldTag.tags.rw == 0) {
// Assert the RW count is consistent with the lock state
fprintf(stderr, "EMSreleaseRW: locked but Count already 0\n");
return -1;
} else {
// Decrement the RW reference count
newTag.tags.rw--;
// If this is the last reader, set the FE tag back to full
if (newTag.tags.rw == 0) { newTag.tags.fe = EMS_TAG_FULL; }
// Attempt to commit the RW reference count & FE tag
if (__sync_bool_compare_and_swap(&(bufTags[EMSdataTag(idx)].byte), oldTag.byte, newTag.byte)) {
return (int) newTag.tags.rw;
} else {
// Another thread decremented the RW count while we also tried
}
}
} else {
if (oldTag.tags.fe != EMS_TAG_BUSY) {
// Assert the RW lock being release is not in some other state then RW_LOCK or BUSY
fprintf(stderr, "EMSreleaseRW: The RW lock being released is in some other state then RW_LOCK or BUSY\n");
return -1;
}
}
// Failed to update the RW count, sleep and retry
NANOSLEEP;
}
}
//==================================================================
// Write EMS honoring the F/E tags
bool EMSwriteUsingTags(int mmapID,
EMSvalueType *key,
EMSvalueType *value,
unsigned char initialFE, // Block until F/E tags are this value
unsigned char finalFE) // Set the tag to this value when done
{
RESET_NAP_TIME;
char *emsBuf = emsBufs[mmapID];
volatile EMStag_t *bufTags = (EMStag_t *) emsBuf;
volatile int64_t *bufInt64 = (int64_t *) emsBuf;
volatile double *bufDouble = (double *) emsBuf;
char *bufChar = emsBuf;
EMStag_t newTag, oldTag, memTag;
int64_t idx = EMSwriteIndexMap(mmapID, key);
if (idx < 0) {
fprintf(stderr, "EMSwriteUsingTags: index out of bounds\n");
return false;
}
// Wait for the memory to be in the initial F/E state and transition to Busy
if (initialFE != EMS_TAG_ANY) {
volatile EMStag_t *maptag;
if (EMSisMapped) { maptag = &bufTags[EMSmapTag(idx)]; }
else { maptag = NULL; }
EMStransitionFEtag(&bufTags[EMSdataTag(idx)], maptag,
initialFE, EMS_TAG_BUSY, EMS_TAG_ANY);
}
while (true) {
idx = idx % bufInt64[EMScbData(EMS_ARR_NELEM)];
memTag.byte = bufTags[EMSdataTag(idx)].byte;
// Wait until FE tag is not BUSY
if (initialFE != EMS_TAG_ANY || finalFE == EMS_TAG_ANY || memTag.tags.fe != EMS_TAG_BUSY) {
oldTag.byte = memTag.byte;
newTag.byte = memTag.byte;
if (finalFE != EMS_TAG_ANY) newTag.tags.fe = EMS_TAG_BUSY;
// Transition FE from !BUSY to BUSY
if (initialFE != EMS_TAG_ANY || finalFE == EMS_TAG_ANY ||
__sync_bool_compare_and_swap(&(bufTags[EMSdataTag(idx)].byte), oldTag.byte, newTag.byte)) {
// If the old data was a string, free it because it will be overwritten
if (oldTag.tags.type == EMS_TYPE_STRING || oldTag.tags.type == EMS_TYPE_JSON) {
EMS_FREE(bufInt64[EMSdataData(idx)]);
}
// Store argument value into EMS memory
switch (value->type) {
case EMS_TYPE_BOOLEAN:
bufInt64[EMSdataData(idx)] = (int64_t) value->value;
break;
case EMS_TYPE_INTEGER:
bufInt64[EMSdataData(idx)] = (int64_t) value->value;
break;
case EMS_TYPE_FLOAT: {
EMSulong_double alias;
alias.u64 = (uint64_t) value->value;
bufDouble[EMSdataData(idx)] = alias.d;
}
break;
case EMS_TYPE_JSON:
case EMS_TYPE_STRING: {
int64_t textOffset;
EMS_ALLOC(textOffset, value->length + 1, bufChar, // NULL padding at end
"EMSwriteUsingTags: out of memory to store string", false);
bufInt64[EMSdataData(idx)] = textOffset;
strcpy(EMSheapPtr(textOffset), (const char *) value->value);
}
break;
case EMS_TYPE_UNDEFINED:
bufInt64[EMSdataData(idx)] = 0xdeadbeef;
break;
default:
fprintf(stderr, "EMSwriteUsingTags: Unknown arg type\n");
return false;
}
oldTag.byte = newTag.byte;
if (finalFE != EMS_TAG_ANY) {
newTag.tags.fe = finalFE;
newTag.tags.rw = 0;
}
newTag.tags.type = value->type;
if (finalFE != EMS_TAG_ANY && bufTags[EMSdataTag(idx)].byte != oldTag.byte) {
fprintf(stderr, "EMSwriteUsingTags: Lost tag lock while BUSY\n");
return false;
}
// Set the tags for the data (and map, if used) back to full to finish the operation
bufTags[EMSdataTag(idx)].byte = newTag.byte;
if (EMSisMapped) bufTags[EMSmapTag(idx)].tags.fe = EMS_TAG_FULL;
return true;
} else {
// Tag was marked BUSY between test read and CAS, must retry
}
} else {
// Tag was already marked BUSY, must retry
}
// Failed to set the tags, sleep and retry
NANOSLEEP;
}
}
//==================================================================
// WriteXF
bool EMSwriteXF(int mmapID, EMSvalueType *key, EMSvalueType *value) {
return EMSwriteUsingTags(mmapID, key, value, EMS_TAG_ANY, EMS_TAG_FULL);
}
//==================================================================
// WriteXE
bool EMSwriteXE(int mmapID, EMSvalueType *key, EMSvalueType *value) {
return EMSwriteUsingTags(mmapID, key, value, EMS_TAG_ANY, EMS_TAG_EMPTY);
}
//==================================================================
// WriteEF
bool EMSwriteEF(int mmapID, EMSvalueType *key, EMSvalueType *value) {
return EMSwriteUsingTags(mmapID, key, value, EMS_TAG_EMPTY, EMS_TAG_FULL);
}
//==================================================================
// Write
bool EMSwrite(int mmapID, EMSvalueType *key, EMSvalueType *value) {
return EMSwriteUsingTags(mmapID, key, value, EMS_TAG_ANY, EMS_TAG_ANY);
}
//==================================================================
// Set only the Full/Empty tag from JavaScript
// without inspecting or modifying the data.
bool EMSsetTag(int mmapID, EMSvalueType *key, bool is_full) {
void *emsBuf = emsBufs[mmapID];
volatile EMStag_t *bufTags = (EMStag_t *) emsBuf;
volatile int64_t *bufInt64 = (int64_t *) emsBuf;
EMStag_t tag;
int64_t idx = EMSkey2index(emsBuf, key, EMSisMapped);
if (idx < 0 || idx >= bufInt64[EMScbData(EMS_ARR_NELEM)]) {
return false;
}
tag.byte = bufTags[EMSdataTag(idx)].byte;
if (is_full) {
tag.tags.fe = EMS_TAG_FULL;
} else {
tag.tags.fe = EMS_TAG_EMPTY;
}
bufTags[EMSdataTag(idx)].byte = tag.byte;
return true;
}
//==================================================================
// Release all the resources associated with an EMS array
bool EMSdestroy(int mmapID, bool do_unlink) {
void *emsBuf = emsBufs[mmapID];
if(munmap(emsBuf, emsBufLengths[mmapID]) != 0) {
fprintf(stderr, "EMSdestroy: Unable to unmap memory\n");
return false;
}
if (do_unlink) {
if (unlink(emsBufFilenames[mmapID]) != 0) {
fprintf(stderr, "EMSdestroy: Unable to unlink file\n");
return false;
}
}
emsBufFilenames[mmapID][0] = 0;
emsBufLengths[mmapID] = 0;
emsBufs[mmapID] = NULL;
return true;
}
//==================================================================
// Return the key of a mapped object given the EMS index
bool EMSindex2key(int mmapID, int64_t idx, EMSvalueType *key) {
void *emsBuf = emsBufs[mmapID];
volatile int64_t *bufInt64 = (int64_t *) emsBuf;
char *bufChar = (char *) emsBuf;
volatile EMStag_t *bufTags = (EMStag_t *) emsBuf;
volatile double *bufDouble = (double *) emsBuf;
if(!EMSisMapped) {
fprintf(stderr, "EMSindex2key: Unmapping an index but Array is not mapped\n");
return false;
}
if (idx < 0 || idx >= bufInt64[EMScbData(EMS_ARR_NELEM)]) {
fprintf(stderr, "EMSindex2key: index out of bounds\n");
return false;
}
key->type = bufTags[EMSmapTag(idx)].tags.type;
switch (key->type) {
case EMS_TYPE_BOOLEAN:
case EMS_TYPE_INTEGER: {
key->value = (void *) bufInt64[EMSmapData(idx)];
return true;
}
case EMS_TYPE_FLOAT: {
EMSulong_double alias;
alias.d = bufDouble[EMSmapData(idx)];
key->value = (void *) alias.u64;
return true;
}
case EMS_TYPE_JSON:
case EMS_TYPE_STRING: {
key->value = (void *)(EMSheapPtr(bufInt64[EMSmapData(idx)]));
return true;
}
case EMS_TYPE_UNDEFINED: {
key->value = NULL;
return true;
}
default:
fprintf(stderr, "EMSindex2key unknown type\n");
return false;
}
}
//==================================================================
// Synchronize the EMS memory to persistent storage
//
bool EMSsync(int mmapID) {
// resultIdx = msync((void*) emsBuf, pgsize, flags);
printf("EMSsync() was called but stubbed out\n");
return false;
}
//==================================================================
// EMS Entry Point: Allocate and initialize the EMS domain memory
//
int EMSinitialize(int64_t nElements, // 0
size_t heapSize, // 1
bool useMap, // 2
const char *filename, // 3
bool persist, // 4
bool useExisting, // 5
bool doDataFill, // 6
bool fillIsJSON, // 7
EMSvalueType *fillValue,// 8
bool doSetFEtags, // 9
bool setFEtagsFull, // 10
int EMSmyIDarg, // 11
bool pinThreads, // 12
int32_t nThreads, // 13
int32_t pctMLock ) { // 14
int fd;
EMSmyID = EMSmyIDarg;
// Node 0 is first and always has mutual exclusion during initialization
// perform once-only initialization here
if (EMSmyID == 0) {
if (!useExisting) {
unlink(filename);
shm_unlink(filename);
}
}
if (useExisting) {
struct timespec sleep_time;
sleep_time.tv_sec = 0;
sleep_time.tv_nsec = 200000000;
struct stat statbuf;
while (stat(filename, &statbuf) != 0) nanosleep(&sleep_time, NULL); // TODO: timeout?
}
if (persist)
fd = open(filename, O_APPEND | O_CREAT | O_RDWR, S_IRUSR | S_IWUSR);
else
fd = shm_open(filename, O_CREAT | O_RDWR, S_IRUSR | S_IWUSR);
if (fd < 0) {
perror("Error opening shared memory file -- Possibly need to be root?");
return -1;
}
size_t nMemBlocks = (heapSize / EMS_MEM_BLOCKSZ) + 1;
size_t nMemBlocksPow2 = emsNextPow2((int64_t) nMemBlocks);
int32_t nMemLevels = __builtin_ctzl(nMemBlocksPow2);
size_t bottomOfMalloc;
size_t filesize;
size_t bottomOfMap = (size_t)EMSdataTagWord(nElements) + (size_t)EMSwordSize; // Map begins 1 word AFTER the last tag word of data
if (useMap) {
bottomOfMalloc = bottomOfMap + bottomOfMap;
} else {
bottomOfMalloc = bottomOfMap;
}
size_t bottomOfHeap = bottomOfMalloc + sizeof(struct emsMem) + (nMemBlocksPow2 * 2 - 2);
if (nElements <= 0) {
filesize = EMS_CB_LOCKS + nThreads; // EMS Control Block
filesize *= sizeof(int);
} else {
filesize = bottomOfHeap + (nMemBlocksPow2 * EMS_MEM_BLOCKSZ);
}
if (ftruncate(fd, (off_t) filesize) != 0) {
if (errno != EINVAL) {
fprintf(stderr, "EMSinitialize: Error during initialization, unable to set memory size to %" PRIu64 " bytes\n",
(uint64_t) filesize);
return -1;
}
}
char *emsBuf = (char *) mmap(0, filesize, PROT_READ | PROT_WRITE, MAP_SHARED, fd, (off_t) 0);
if (emsBuf == MAP_FAILED) {
fprintf(stderr, "EMSinitialize: Unable to map domain memory\n");
return -1;
}
close(fd);
if (nElements <= 0) pctMLock = 100; // lock RAM if master control block
if (mlock((void *) emsBuf, (size_t) (filesize * (pctMLock / 100))) != 0) {
fprintf(stderr, "EMSinitialize NOTICE: EMS thread %d was not able to lock EMS memory to RAM for %s\n", EMSmyID, filename);
} else {
// success
}
volatile int64_t *bufInt64 = (int64_t *) emsBuf;
volatile double *bufDouble = (double *) emsBuf;
char *bufChar = emsBuf;
volatile int *bufInt32 = (int32_t *) emsBuf;
volatile EMStag_t *bufTags = (EMStag_t *) emsBuf;
if (EMSmyID == 0) {
if (nElements <= 0) { // This is the EMS CB
bufInt32[EMS_CB_NTHREADS] = nThreads;
bufInt32[EMS_CB_NBAR0] = nThreads;
bufInt32[EMS_CB_NBAR1] = nThreads;
bufInt32[EMS_CB_BARPHASE] = 0;
bufInt32[EMS_CB_CRITICAL] = 0;
bufInt32[EMS_CB_SINGLE] = 0;
for (int i = EMS_CB_LOCKS; i < EMS_CB_LOCKS + nThreads; i++) {
bufInt32[i] = EMS_TAG_BUSY; // Reset all locks
}
} else { // This is a user data domain
if (!useExisting) {
EMStag_t tag;
tag.tags.rw = 0;
tag.tags.type = EMS_TYPE_INTEGER;
tag.tags.fe = EMS_TAG_FULL;
bufInt64[EMScbData(EMS_ARR_NELEM)] = nElements;
bufInt64[EMScbData(EMS_ARR_HEAPSZ)] = heapSize; // Unused?
bufInt64[EMScbData(EMS_ARR_MAPBOT)] = bottomOfMap / EMSwordSize;
bufInt64[EMScbData(EMS_ARR_MALLOCBOT)] = bottomOfMalloc;
bufInt64[EMScbData(EMS_ARR_HEAPBOT)] = bottomOfHeap;
bufInt64[EMScbData(EMS_ARR_Q_BOTTOM)] = 0;
bufTags[EMScbTag(EMS_ARR_Q_BOTTOM)].byte = tag.byte;
bufInt64[EMScbData(EMS_ARR_STACKTOP)] = 0;
bufTags[EMScbTag(EMS_ARR_STACKTOP)].byte = tag.byte;
bufInt64[EMScbData(EMS_ARR_MEM_MUTEX)] = EMS_TAG_EMPTY;
bufInt64[EMScbData(EMS_ARR_FILESZ)] = filesize;
struct emsMem *emsMemBuffer = (struct emsMem *) &bufChar[bufInt64[EMScbData(EMS_ARR_MALLOCBOT)]];
emsMemBuffer->level = nMemLevels;
}
}
}
if (pinThreads) {
#if defined(__linux)
cpu_set_t cpuset;
CPU_ZERO(&cpuset);
CPU_SET((EMSmyID % nThreads), &cpuset); // Round-robin over-subscribed systems
sched_setaffinity(0, sizeof(cpu_set_t), &cpuset);
#endif
}
EMStag_t tag;
tag.tags.rw = 0;
int64_t iterPerThread = (nElements / nThreads) + 1;
int64_t startIter = iterPerThread * EMSmyID;
int64_t endIter = iterPerThread * (EMSmyID + 1);
size_t fillStrLen = 0;
if (doDataFill && (fillValue->type == EMS_TYPE_JSON || fillValue->type == EMS_TYPE_STRING)) {
fillStrLen = fillValue->length;
}
if (endIter > nElements) endIter = nElements;
for (int64_t idx = startIter; idx < endIter; idx++) {
tag.tags.rw = 0;
if (doDataFill) {
tag.tags.type = fillValue->type;
switch (tag.tags.type) {
case EMS_TYPE_BOOLEAN:
case EMS_TYPE_INTEGER:
bufInt64[EMSdataData(idx)] = (int64_t) fillValue->value;
break;
case EMS_TYPE_FLOAT: {
EMSulong_double alias;
alias.u64 = (uint64_t) fillValue->value;
bufDouble[EMSdataData(idx)] = alias.d;
}
break;
case EMS_TYPE_UNDEFINED:
bufInt64[EMSdataData(idx)] = 0xdeadbeef;
break;
case EMS_TYPE_JSON:
case EMS_TYPE_STRING: {
int64_t textOffset;
EMS_ALLOC(textOffset, fillStrLen + 1, bufChar,
"EMSinitialize: out of memory to store string", false);
bufInt64[EMSdataData(idx)] = textOffset;
strcpy(EMSheapPtr(textOffset), (const char *) fillValue->value);
}
break;
default:
fprintf(stderr, "EMSinitialize: fill type is unknown\n");
return -1;
}
} else {
tag.tags.type = EMS_TYPE_UNDEFINED;
}
if (doSetFEtags) {
if (setFEtagsFull) tag.tags.fe = EMS_TAG_FULL;
else tag.tags.fe = EMS_TAG_EMPTY;
}
if (doSetFEtags || doDataFill) {
bufTags[EMSdataTag(idx)].byte = tag.byte;
}
if (useMap) {
bufTags[EMSmapTag(idx)].tags.fe = EMS_TAG_FULL;
if (!useExisting) { bufTags[EMSmapTag(idx)].tags.type = EMS_TYPE_UNDEFINED; }
}
}
int emsBufN = 0;
while(emsBufN < EMS_MAX_N_BUFS && emsBufs[emsBufN] != NULL) emsBufN++;
if(emsBufN < EMS_MAX_N_BUFS) {
emsBufs[emsBufN] = emsBuf;
emsBufLengths[emsBufN] = filesize;
strncpy(emsBufFilenames[emsBufN], filename, MAX_FNAME_LEN);
} else {
fprintf(stderr, "EMSinitialize: ERROR - Unable to allocate a buffer ID/index\n");
emsBufN = -1;
}
return emsBufN;
}
================================================
FILE: src/ems.h
================================================
/*-----------------------------------------------------------------------------+
| Extended Memory Semantics (EMS) Version 1.4.0 |
| Synthetic Semantics http://www.synsem.com/ mogill@synsem.com |
+-----------------------------------------------------------------------------+
| Copyright (c) 2011-2014, Synthetic Semantics LLC. All rights reserved. |
| Copyright (c) 2015-2016, Jace A Mogill. All rights reserved. |
| |
| Redistribution and use in source and binary forms, with or without |
| modification, are permitted provided that the following conditions are met: |
| * Redistributions of source code must retain the above copyright |
| notice, this list of conditions and the following disclaimer. |
| * Redistributions in binary form must reproduce the above copyright |
| notice, this list of conditions and the following disclaimer in the |
| documentation and/or other materials provided with the distribution. |
| * Neither the name of the Synthetic Semantics nor the names of its |
| contributors may be used to endorse or promote products derived |
| from this software without specific prior written permission. |
| |
| THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS |
| "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT |
| LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR |
| A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL SYNTHETIC |
| SEMANTICS LLC BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, |
| EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, |
| PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR |
| PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF |
| LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING |
| NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS |
| SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. |
| |
+-----------------------------------------------------------------------------*/
#ifndef EMSPROJ_EMS_H
#define EMSPROJ_EMS_H
#define __STDC_FORMAT_MACROS 1
#include
#include
#include
#include
#include
#include
#include
#include
#include
#include
#include
#include
#if !defined _GNU_SOURCE
# define _GNU_SOURCE
#endif
#include
#include "ems_alloc.h"
//==================================================================
// EMS Full/Empty Tag States
//
#define EMS_TAG_ANY ((unsigned char)4) // Never stored, used for matching
#define EMS_TAG_RW_LOCK ((unsigned char)3)
#define EMS_TAG_BUSY ((unsigned char)2)
#define EMS_TAG_EMPTY ((unsigned char)1)
#define EMS_TAG_FULL ((unsigned char)0)
//==================================================================
// EMS Data types
//
#define EMS_TYPE_INVALID ((unsigned char)0)
#define EMS_TYPE_BOOLEAN ((unsigned char)1)
#define EMS_TYPE_STRING ((unsigned char)2)
#define EMS_TYPE_FLOAT ((unsigned char)3)
#define EMS_TYPE_INTEGER ((unsigned char)4)
#define EMS_TYPE_UNDEFINED ((unsigned char)5)
#define EMS_TYPE_JSON ((unsigned char)6) // Catch-all for JSON arrays and Objects
//==================================================================
// Control Block layout stored at the head of each EMS array
//
// Name Offset Description of contents
//---------------------------------------------------------------------------------------
#define NWORDS_PER_CACHELINE 16
#define EMS_ARR_NELEM (0 * NWORDS_PER_CACHELINE) // Maximum number of elements in the EMS array
#define EMS_ARR_HEAPSZ (1 * NWORDS_PER_CACHELINE) // # bytes of storage for array data: strings, JSON, maps, etc.
#define EMS_ARR_Q_BOTTOM (2 * NWORDS_PER_CACHELINE) // Current index of the queue bottom
#define EMS_ARR_STACKTOP (3 * NWORDS_PER_CACHELINE) // Current index of the top of the stack/queue
#define EMS_ARR_MAPBOT (4 * NWORDS_PER_CACHELINE) // Index of the base of the index map
#define EMS_ARR_MALLOCBOT (5 * NWORDS_PER_CACHELINE) // Index of the base of the heap -- malloc structs start here
#define EMS_ARR_HEAPBOT (6 * NWORDS_PER_CACHELINE) // Index of the base of data on the heap -- strings start here
#define EMS_ARR_MEM_MUTEX (7 * NWORDS_PER_CACHELINE) // Mutex lock for thememory allocator of this EMS region's
#define EMS_ARR_FILESZ (8 * NWORDS_PER_CACHELINE) // Total size in bytes of the EMS region
// Tag data may follow data by as much as 8 words, so
// A gap of at least 8 words is required to leave space for
// the tags associated with header data
#define EMS_ARR_CB_SIZE (16 * NWORDS_PER_CACHELINE) // Index of the first EMS array element
//==================================================================
// EMS Control Block -- Global State for EMS
#define EMS_CB_NTHREADS 0 // Number of threads
#define EMS_CB_NBAR0 1 // Number of threads at Barrier 0
#define EMS_CB_NBAR1 2 // Number of threads at Barrier 1
#define EMS_CB_BARPHASE 3 // Current Barrier Phase (0 or 1)
#define EMS_CB_CRITICAL 4 // Mutex for critical regions
#define EMS_CB_SINGLE 5 // Number of threads passed through an execute-once region
#define EMS_LOOP_IDX 6 // Index of next iteration in a Parallel loop to schedule
#define EMS_LOOP_START 7 // Index of first iteration in a parallel loop
#define EMS_LOOP_END 8 // Index of last iteration in a parallel loop
#define EMS_LOOP_CHUNKSZ 9 // Current Number of iterations per thread
#define EMS_LOOP_MINCHUNK 10 // Smallest number of iterations per thread
#define EMS_LOOP_SCHED 11 // Parallel loop scheduling method:
#define EMS_SCHED_GUIDED 1200
#define EMS_SCHED_DYNAMIC 1201
#define EMS_SCHED_STATIC 1202
#define EMS_CB_LOCKS 12 // First index of an array of locks, one lock per thread
//==================================================================
// Pointers to mmapped EMS buffers
#define EMS_MAX_N_BUFS 4096
#define MAX_NUMBER2STR_LEN 40 // Maximum number of characters in a %d or %f format
#define MAX_FNAME_LEN 256
extern char *emsBufs[EMS_MAX_N_BUFS];
extern size_t emsBufLengths[EMS_MAX_N_BUFS];
extern char emsBufFilenames[EMS_MAX_N_BUFS][MAX_FNAME_LEN];
// Maximum number of slots to check due to conflicts
#define MAX_OPEN_HASH_STEPS 200
//==================================================================
// Macros to translate from EMS Data Indexes and EMS Control Block
// indexes to offsets in the EMS shared memory
#define EMSwordSize (sizeof(size_t))
#define EMSnWordsPerTagWord (EMSwordSize-1)
#define EMSnWordsPerLine EMSwordSize
//==================================================================
// Layout of EMS memory
// Tagged Memory
// CB: Control Block of array state
// Data: Scalar user data
// Map: Scalar index map data
// Untagged Memory
// Malloc: Storage for the free/used structures
// Heap: Open Heap storage
#define EMSappIdx2emsIdx(idx) ((((idx) / EMSnWordsPerTagWord) * EMSnWordsPerLine) + ((idx) % EMSnWordsPerTagWord) )
#define EMSappIdx2LineIdx(idx) ( ((idx) / EMSnWordsPerTagWord) * EMSnWordsPerLine)
#define EMSappIdx2TagWordIdx(idx) ( EMSappIdx2LineIdx(idx) + EMSnWordsPerTagWord )
#define EMSappIdx2TagWordOffset(idx) ( EMSappIdx2TagWordIdx(idx) * EMSwordSize )
#define EMSappTag2emsTag(idx) ( EMSappIdx2TagWordOffset(idx) + ((idx) % EMSnWordsPerTagWord) )
#define EMScbData(idx) EMSappIdx2emsIdx(idx)
#define EMScbTag(idx) EMSappTag2emsTag(idx)
#define EMSdataData(idx) ( EMSappIdx2emsIdx((idx) + EMS_ARR_CB_SIZE) )
#define EMSdataTag(idx) ( EMSappTag2emsTag((idx) + EMS_ARR_CB_SIZE) )
#define EMSdataTagWord(idx) ( EMSappIdx2TagWordOffset((idx) + EMS_ARR_CB_SIZE) )
#define EMSmapData(idx) ( EMSappIdx2emsIdx((idx) + EMS_ARR_CB_SIZE + bufInt64[EMScbData(EMS_ARR_NELEM)]) )
#define EMSmapTag(idx) ( EMSappTag2emsTag((idx) + EMS_ARR_CB_SIZE + bufInt64[EMScbData(EMS_ARR_NELEM)]) )
#define EMSheapPtr(idx) ( &bufChar[ bufInt64[EMScbData(EMS_ARR_HEAPBOT)] + (idx) ] )
#define EMS_MEM_MALLOCBOT(bufChar) ((struct emsMem *) &bufChar[ bufInt64[EMScbData(EMS_ARR_MALLOCBOT)] ])
//==================================================================
// Yield the processor and sleep (using exponential decay) without
// using resources/
// Used within spin-loops to reduce hot-spotting
#define RESET_NAP_TIME int EMScurrentNapTime = 1
#define MAX_NAP_TIME 1000000
#define NANOSLEEP { \
struct timespec sleep_time; \
sleep_time.tv_sec = 0; \
sleep_time.tv_nsec = EMScurrentNapTime; \
nanosleep(&sleep_time, NULL); \
EMScurrentNapTime *= 2; \
if(EMScurrentNapTime > MAX_NAP_TIME) { \
EMScurrentNapTime = MAX_NAP_TIME; \
} \
}
#define EMS_ALLOC(addr, len, bufChar, errmsg, retval) \
addr = emsMutexMem_alloc( EMS_MEM_MALLOCBOT(bufChar), \
(size_t) len, (char*) &bufInt64[EMScbData(EMS_ARR_MEM_MUTEX)] ); \
if(addr < 0) { \
fprintf(stderr, "%s:%d (%s) ERROR: EMS memory allocation of len(%zx) failed: %s\n", \
__FILE__, __LINE__, __FUNCTION__, len, errmsg); \
return retval; \
}
#define EMS_FREE(addr) \
emsMutexMem_free( EMS_MEM_MALLOCBOT(bufChar), \
(size_t) addr, (char*) &bufInt64[EMScbData(EMS_ARR_MEM_MUTEX)] )
size_t emsMutexMem_alloc(struct emsMem *heap, // Base of EMS malloc structs
size_t len, // Number of bytes to allocate
volatile char *mutex); // Pointer to the mem allocator's mutex
void emsMutexMem_free(struct emsMem *heap, // Base of EMS malloc structs
size_t addr, // Offset of alloc'd block in EMS memory
volatile char *mutex); // Pointer to the mem allocator's mutex
extern int EMSmyID; // EMS Thread ID
#define EMSisMapped (bufInt64[EMScbData(EMS_ARR_MAPBOT)]*(int64_t)EMSwordSize != bufInt64[EMScbData(EMS_ARR_MALLOCBOT)])
#include "ems_proto.h"
#endif //EMSPROJ_EMS_H
================================================
FILE: src/ems_alloc.cc
================================================
/*-----------------------------------------------------------------------------+
| Extended Memory Semantics (EMS) Version 1.3.0 |
| Synthetic Semantics http://www.synsem.com/ mogill@synsem.com |
+-----------------------------------------------------------------------------+
| Copyright (c) 2011-2014, Synthetic Semantics LLC. All rights reserved. |
| Copyright (c) 2015-2016, Jace A Mogill. All rights reserved. |
| |
| Redistribution and use in source and binary forms, with or without |
| modification, are permitted provided that the following conditions are met: |
| * Redistributions of source code must retain the above copyright |
| notice, this list of conditions and the following disclaimer. |
| * Redistributions in binary form must reproduce the above copyright |
| notice, this list of conditions and the following disclaimer in the |
| documentation and/or other materials provided with the distribution. |
| * Neither the name of the Synthetic Semantics nor the names of its |
| contributors may be used to endorse or promote products derived |
| from this software without specific prior written permission. |
| |
| THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS |
| "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT |
| LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR |
| A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL SYNTHETIC |
| SEMANTICS LLC BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, |
| EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, |
| PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR |
| PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF |
| LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING |
| NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS |
| SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. |
| |
+-----------------------------------------------------------------------------*/
#include "ems_alloc.h"
#include
#include
#include
#define BUDDY_UNUSED 0
#define BUDDY_USED 1
#define BUDDY_SPLIT 2
#define BUDDY_FULL 3
//-----------------------------------------------------------------------------+
// Allocate memory for testing --
// Performed as part of new EMS object initialization
struct emsMem *emsMem_new(int level) {
size_t size = 1UL << level;
printf("emsMem_new: malloc sz=%ld\n", size);
struct emsMem *self = (struct emsMem *) malloc(sizeof(struct emsMem) + sizeof(uint8_t) * (size * 2 - 2));
self->level = level;
memset(self->tree, BUDDY_UNUSED, size * 2 - 1);
return self;
}
//-----------------------------------------------------------------------------+
// De-allocate memory from testing, not part of EMS object
void emsMem_delete(struct emsMem *self) {
free(self);
}
//-----------------------------------------------------------------------------+
// Pow-2 utility functions for 64 bit
size_t emsNextPow2(int64_t x) {
if (__builtin_popcountll(x) == 1) return x;
// Yes, that will overflow on 63 bit numbers. Hopefully someone will rewrite this by then.
// fprintf(stderr, ">>>lz=%d shift=%d\n", __builtin_clzl(x), (64 - __builtin_clzl(x)));
return (1UL << (64 - __builtin_clzll(x)));
}
static inline int64_t EMS_index_offset(int64_t index, int32_t level, int64_t max_level) {
return ((index + 1) - (1UL << level)) << (max_level - level);
}
//-----------------------------------------------------------------------------+
// Mark the parent buddy of this buddy
static void EMS_mark_parent(struct emsMem *self, int64_t index) {
for (; ;) {
int64_t buddy = index - 1 + (index & 1) * 2;
if (buddy > 0 && (self->tree[buddy] == BUDDY_USED || self->tree[buddy] == BUDDY_FULL)) {
index = (index + 1) / 2 - 1;
self->tree[index] = BUDDY_FULL;
} else {
return;
}
}
}
//-----------------------------------------------------------------------------+
// Allocate new memory from the EMS heap
size_t emsMem_alloc(struct emsMem *self, size_t bytesRequested) {
size_t size;
size = emsNextPow2((bytesRequested + (EMS_MEM_BLOCKSZ - 1)) / EMS_MEM_BLOCKSZ);
if (size == 0) size++;
size_t length = 1UL << self->level;
// fprintf(stderr, "emsMem_alloc: self=%x size=%ld s=%ld len=%ld\n", self, size, s, length);
if (size > length) return -1;
int64_t index = 0;
int32_t level = 0;
while (index >= 0) {
if (size == length) {
if (self->tree[index] == BUDDY_UNUSED) {
self->tree[index] = BUDDY_USED;
EMS_mark_parent(self, index);
return ((size_t)EMS_index_offset(index, level, self->level) * EMS_MEM_BLOCKSZ);
}
} else {
// size < length
switch (self->tree[index]) {
case BUDDY_USED:
case BUDDY_FULL:
break;
case BUDDY_UNUSED:
// split first
self->tree[index] = BUDDY_SPLIT;
self->tree[index * 2 + 1] = BUDDY_UNUSED;
self->tree[index * 2 + 2] = BUDDY_UNUSED;
default:
index = index * 2 + 1;
length /= 2;
level++;
continue;
}
}
if (index & 1) {
++index;
continue;
}
for (; ;) {
level--;
length *= 2;
index = (index + 1) / 2 - 1;
if (index < 0)
return -1;
if (index & 1) {
++index;
break;
}
}
}
return -1;
}
//-----------------------------------------------------------------------------+
// Combine two buddies into one node
static void EMS_combine(struct emsMem *self, int64_t index) {
for (; ;) {
int64_t buddy = index - 1 + (index & 1) * 2;
if (buddy < 0 || self->tree[buddy] != BUDDY_UNUSED) {
self->tree[index] = BUDDY_UNUSED;
while (((index = (index + 1) / 2 - 1) >= 0) && self->tree[index] == BUDDY_FULL) {
self->tree[index] = BUDDY_SPLIT;
}
return;
}
index = (index + 1) / 2 - 1;
}
}
//-----------------------------------------------------------------------------+
// Release EMS memory back to the heap for reuse
void emsMem_free(struct emsMem *self, size_t offset) {
offset /= EMS_MEM_BLOCKSZ;
assert(offset < (1UL << self->level));
size_t left = 0;
size_t length = 1UL << self->level;
int64_t index = 0;
for (; ;) {
switch (self->tree[index]) {
case BUDDY_USED:
assert(offset == left);
EMS_combine(self, index);
return;
case BUDDY_UNUSED:
assert(0);
return;
default:
length /= 2;
if (offset < left + length) {
index = index * 2 + 1;
} else {
left += length;
index = index * 2 + 2;
}
break;
}
}
}
//-----------------------------------------------------------------------------+
// Return the size of a block of memory
size_t emsMem_size(struct emsMem *self, size_t offset) {
assert(offset < (1UL << self->level));
size_t left = 0;
size_t length = 1UL << self->level;
size_t index = 0;
for (; ;) {
switch (self->tree[index]) {
case BUDDY_USED:
assert(offset == left);
return length * EMS_MEM_BLOCKSZ;
case BUDDY_UNUSED:
assert(0);
return length * EMS_MEM_BLOCKSZ;
default:
length /= 2;
if (offset < left + length) {
index = index * 2 + 1;
} else {
left += length;
index = index * 2 + 2;
}
break;
}
}
}
//-----------------------------------------------------------------------------+
// Diagnostic state dump
static void EMS_dump(struct emsMem *self, size_t index, int32_t level) {
switch (self->tree[index]) {
case BUDDY_UNUSED:
printf("(%lld:%ld)", (long long int) EMS_index_offset(index, level, self->level),
1UL << (self->level - level));
break;
case BUDDY_USED:
printf("[%lld:%ld]", (long long int) EMS_index_offset(index, level, self->level),
1UL << (self->level - level));
break;
case BUDDY_FULL:
printf("{");
EMS_dump(self, index * 2 + 1, level + 1);
EMS_dump(self, index * 2 + 2, level + 1);
printf("}");
break;
default:
printf("(");
EMS_dump(self, index * 2 + 1, level + 1);
EMS_dump(self, index * 2 + 2, level + 1);
printf(")");
break;
}
}
void emsMem_dump(struct emsMem *self) {
EMS_dump(self, 0, 0);
printf("\n");
}
================================================
FILE: src/ems_alloc.h
================================================
/*-----------------------------------------------------------------------------+
| Extended Memory Semantics (EMS) Version 1.4.0 |
| Synthetic Semantics http://www.synsem.com/ mogill@synsem.com |
+-----------------------------------------------------------------------------+
| Copyright (c) 2011-2014, Synthetic Semantics LLC. All rights reserved. |
| Copyright (c) 2015-2016, Jace A Mogill. All rights reserved. |
| |
| Redistribution and use in source and binary forms, with or without |
| modification, are permitted provided that the following conditions are met: |
| * Redistributions of source code must retain the above copyright |
| notice, this list of conditions and the following disclaimer. |
| * Redistributions in binary form must reproduce the above copyright |
| notice, this list of conditions and the following disclaimer in the |
| documentation and/or other materials provided with the distribution. |
| * Neither the name of the Synthetic Semantics nor the names of its |
| contributors may be used to endorse or promote products derived |
| from this software without specific prior written permission. |
| |
| THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS |
| "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT |
| LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR |
| A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL SYNTHETIC |
| SEMANTICS LLC BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, |
| EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, |
| PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR |
| PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF |
| LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING |
| NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS |
| SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. |
| |
+-----------------------------------------------------------------------------*/
#ifndef EMS_MEMORY_ALLOCATION_H
#define EMS_MEMORY_ALLOCATION_H
#include
#include
// The block size used by the memory allocator for allocating heap space.
// May be any positive non-zero value
#define EMS_MEM_BLOCKSZ 32
// Buddy allocator control structure
struct emsMem {
int32_t level;
uint8_t tree[1];
};
struct emsMem *emsMem_new(int level);
void emsMem_delete(struct emsMem *);
size_t emsMem_alloc(struct emsMem *, size_t bytesRequested);
void emsMem_free(struct emsMem *, size_t offset);
size_t emsMem_size(struct emsMem *, size_t offset);
void emsMem_dump(struct emsMem *);
size_t emsNextPow2(int64_t x);
#endif
================================================
FILE: src/ems_proto.h
================================================
/*-----------------------------------------------------------------------------+
| Extended Memory Semantics (EMS) Version 1.5.0 |
| Synthetic Semantics http://www.synsem.com/ mogill@synsem.com |
+-----------------------------------------------------------------------------+
| Copyright (c) 2011-2014, Synthetic Semantics LLC. All rights reserved. |
| Copyright (c) 2015-2017, Jace A Mogill. All rights reserved. |
| |
| Redistribution and use in source and binary forms, with or without |
| modification, are permitted provided that the following conditions are met: |
| * Redistributions of source code must retain the above copyright |
| notice, this list of conditions and the following disclaimer. |
| * Redistributions in binary form must reproduce the above copyright |
| notice, this list of conditions and the following disclaimer in the |
| documentation and/or other materials provided with the distribution. |
| * Neither the name of the Synthetic Semantics nor the names of its |
| contributors may be used to endorse or promote products derived |
| from this software without specific prior written permission. |
| |
| THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS |
| "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT |
| LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR |
| A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL SYNTHETIC |
| SEMANTICS LLC BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, |
| EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, |
| PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR |
| PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF |
| LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING |
| NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS |
| SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. |
| |
+-----------------------------------------------------------------------------*/
#include "ems_types.h"
// ---------------------------------------------------------------------------------
// Non-exposed API functions
int64_t EMSwriteIndexMap(const int mmapID, EMSvalueType *key);
int64_t EMSkey2index(void *emsBuf, EMSvalueType *key, bool is_mapped);
int64_t EMShashString(const char *key);
// ---------------------------------------------------------------------------------
// External API functions
extern "C" int EMScriticalEnter(int mmapID, int timeout);
extern "C" bool EMScriticalExit(int mmapID);
extern "C" int EMSbarrier(int mmapID, int timeout);
extern "C" bool EMSsingleTask(int mmapID);
extern "C" bool EMScas(int mmapID, EMSvalueType *key,
EMSvalueType *oldValue, EMSvalueType *newValue,
EMSvalueType *returnValue);
extern "C" bool EMSfaa(int mmapID, EMSvalueType *key, EMSvalueType *value, EMSvalueType *returnValue);
extern "C" int EMSpush(int mmapID, EMSvalueType *value);
extern "C" bool EMSpop(int mmapID, EMSvalueType *returnValue);
extern "C" int EMSenqueue(int mmapID, EMSvalueType *value);
extern "C" bool EMSdequeue(int mmapID, EMSvalueType *returnValue);
extern "C" bool EMSloopInit(int mmapID, int32_t start, int32_t end, int32_t minChunk, int schedule_mode);
extern "C" bool EMSloopChunk(int mmapID, int32_t *start, int32_t *end);
extern "C" unsigned char EMStransitionFEtag(EMStag_t volatile *tag, EMStag_t volatile *mapTag, unsigned char oldFE, unsigned char newFE, unsigned char oldType);
extern "C" bool EMSreadRW(const int mmapID, EMSvalueType *key, EMSvalueType *returnValue);
extern "C" bool EMSreadFF(const int mmapID, EMSvalueType *key, EMSvalueType *returnValue);
extern "C" bool EMSreadFE(const int mmapID, EMSvalueType *key, EMSvalueType *returnValue);
extern "C" bool EMSread(const int mmapID, EMSvalueType *key, EMSvalueType *returnValue);
extern "C" int EMSreleaseRW(const int mmapID, EMSvalueType *key);
extern "C" bool EMSwriteXF(int mmapID, EMSvalueType *key, EMSvalueType *value);
extern "C" bool EMSwriteXE(int mmapID, EMSvalueType *key, EMSvalueType *value);
extern "C" bool EMSwriteEF(int mmapID, EMSvalueType *key, EMSvalueType *value);
extern "C" bool EMSwrite(int mmapID, EMSvalueType *key, EMSvalueType *value);
extern "C" bool EMSsetTag(int mmapID, EMSvalueType *key, bool is_full);
extern "C" bool EMSdestroy(int mmapID, bool do_unlink);
extern "C" bool EMSindex2key(int mmapID, int64_t idx, EMSvalueType *key);
extern "C" bool EMSsync(int mmapID);
extern "C" int EMSinitialize(int64_t nElements, // 0
size_t heapSize, // 1
bool useMap, // 2
const char *filename, // 3
bool persist, // 4
bool useExisting, // 5
bool doDataFill, // 6
bool fillIsJSON, // 7
EMSvalueType *fillValue, // 8
bool doSetFEtags, // 9
bool setFEtagsFull, // 10
int EMSmyID, // 11
bool pinThreads, // 12
int32_t nThreads, // 13
int32_t pctMLock ); // 14
================================================
FILE: src/ems_types.h
================================================
/*-----------------------------------------------------------------------------+
| Extended Memory Semantics (EMS) Version 1.5.0 |
| Synthetic Semantics http://www.synsem.com/ mogill@synsem.com |
+-----------------------------------------------------------------------------+
| Copyright (c) 2017, Jace A Mogill. All rights reserved. |
| |
| Redistribution and use in source and binary forms, with or without |
| modification, are permitted provided that the following conditions are met: |
| * Redistributions of source code must retain the above copyright |
| notice, this list of conditions and the following disclaimer. |
| * Redistributions in binary form must reproduce the above copyright |
| notice, this list of conditions and the following disclaimer in the |
| documentation and/or other materials provided with the distribution. |
| * Neither the name of the Synthetic Semantics nor the names of its |
| contributors may be used to endorse or promote products derived |
| from this software without specific prior written permission. |
| |
| THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS |
| "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT |
| LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR |
| A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL SYNTHETIC |
| SEMANTICS LLC BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, |
| EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, |
| PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR |
| PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF |
| LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING |
| NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS |
| SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. |
| |
+-----------------------------------------------------------------------------*/
#ifndef EMSPROJ_EMS_TYPES_H
#define EMSPROJ_EMS_TYPES_H
// Bitfields of a Tag Byte
#define EMS_TYPE_NBITS_FE 2
#define EMS_TYPE_NBITS_TYPE 3
#define EMS_TYPE_NBITS_RW 3
#define EMS_RW_NREADERS_MAX ((1 << EMS_TYPE_NBITS_RW) - 1)
typedef union {
struct {
unsigned char fe : EMS_TYPE_NBITS_FE;
unsigned char type : EMS_TYPE_NBITS_TYPE;
unsigned char rw : EMS_TYPE_NBITS_RW;
} tags;
unsigned char byte;
} EMStag_t;
#define EMS_VALUE_TYPE_INITIALIZER {.length=0, .value=NULL, .type=EMS_TYPE_INVALID}
// Type-punning is now a warning in GCC, but this syntax is still okay
typedef union {
double d;
uint64_t u64;
} EMSulong_double;
// Internal EMS representation of a JSON value
typedef struct {
size_t length; // Defined only for JSON and strings
void *value;
unsigned char type;
} EMSvalueType;
#endif
//EMSPROJ_EMS_TYPES_H
================================================
FILE: src/loops.cc
================================================
/*-----------------------------------------------------------------------------+
| Extended Memory Semantics (EMS) Version 1.3.0 |
| Synthetic Semantics http://www.synsem.com/ mogill@synsem.com |
+-----------------------------------------------------------------------------+
| Copyright (c) 2011-2014, Synthetic Semantics LLC. All rights reserved. |
| Copyright (c) 2015-2016, Jace A Mogill. All rights reserved. |
| |
| Redistribution and use in source and binary forms, with or without |
| modification, are permitted provided that the following conditions are met: |
| * Redistributions of source code must retain the above copyright |
| notice, this list of conditions and the following disclaimer. |
| * Redistributions in binary form must reproduce the above copyright |
| notice, this list of conditions and the following disclaimer in the |
| documentation and/or other materials provided with the distribution. |
| * Neither the name of the Synthetic Semantics nor the names of its |
| contributors may be used to endorse or promote products derived |
| from this software without specific prior written permission. |
| |
| THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS |
| "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT |
| LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR |
| A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL SYNTHETIC |
| SEMANTICS LLC BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, |
| EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, |
| PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR |
| PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF |
| LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING |
| NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS |
| SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. |
| |
+-----------------------------------------------------------------------------*/
#include "ems.h"
//==================================================================
// Parallel Loop -- context initialization
//
bool EMSloopInit(int mmapID, int32_t start, int32_t end, int32_t minChunk, int schedule_mode) {
void *emsBuf = emsBufs[mmapID];
int32_t *bufInt32 = (int32_t *) emsBuf;
bool success = true;
bufInt32[EMS_LOOP_IDX] = start;
bufInt32[EMS_LOOP_START] = start;
bufInt32[EMS_LOOP_END] = end;
switch (schedule_mode) {
case EMS_SCHED_GUIDED:
bufInt32[EMS_LOOP_CHUNKSZ] = ((end - start) / 2) / bufInt32[EMS_CB_NTHREADS];
if (bufInt32[EMS_LOOP_CHUNKSZ] < minChunk) bufInt32[EMS_LOOP_CHUNKSZ] = minChunk;
bufInt32[EMS_LOOP_MINCHUNK] = minChunk;
bufInt32[EMS_LOOP_SCHED] = EMS_SCHED_GUIDED;
break;
case EMS_SCHED_DYNAMIC:
bufInt32[EMS_LOOP_CHUNKSZ] = 1;
bufInt32[EMS_LOOP_MINCHUNK] = 1;
bufInt32[EMS_LOOP_SCHED] = EMS_SCHED_DYNAMIC;
break;
default:
fprintf(stderr, "NodeJSloopInit: Unknown schedule modes\n");
success = false;
}
return success;
}
//==================================================================
// Determine the current block of iterations to assign to an
// an idle thread
// JQM TODO BUG -- convert to 64 bit using fe tags
//
bool EMSloopChunk(int mmapID, int32_t *start, int32_t *end) {
void *emsBuf = emsBufs[mmapID];
int32_t *bufInt32 = (int32_t *) emsBuf;
int chunkSize = bufInt32[EMS_LOOP_CHUNKSZ];
*start = __sync_fetch_and_add(&(bufInt32[EMS_LOOP_IDX]), chunkSize);
*end = *start + chunkSize;
if (*start > bufInt32[EMS_LOOP_END]) *end = 0;
if (*end > bufInt32[EMS_LOOP_END]) *end = bufInt32[EMS_LOOP_END];
if (bufInt32[EMS_LOOP_SCHED] == EMS_SCHED_GUIDED) {
// Compute the size of the chunk the next thread should use
int newSz = (int) ((bufInt32[EMS_LOOP_END] - *start) / 2) / bufInt32[EMS_CB_NTHREADS];
if (newSz < bufInt32[EMS_LOOP_MINCHUNK]) newSz = bufInt32[EMS_LOOP_MINCHUNK];
bufInt32[EMS_LOOP_CHUNKSZ] = newSz;
}
return true;
}
================================================
FILE: src/primitives.cc
================================================
/*-----------------------------------------------------------------------------+
| Extended Memory Semantics (EMS) Version 1.3.0 |
| Synthetic Semantics http://www.synsem.com/ mogill@synsem.com |
+-----------------------------------------------------------------------------+
| Copyright (c) 2011-2014, Synthetic Semantics LLC. All rights reserved. |
| Copyright (c) 2015-2016, Jace A Mogill. All rights reserved. |
| |
| Redistribution and use in source and binary forms, with or without |
| modification, are permitted provided that the following conditions are met: |
| * Redistributions of source code must retain the above copyright |
| notice, this list of conditions and the following disclaimer. |
| * Redistributions in binary form must reproduce the above copyright |
| notice, this list of conditions and the following disclaimer in the |
| documentation and/or other materials provided with the distribution. |
| * Neither the name of the Synthetic Semantics nor the names of its |
| contributors may be used to endorse or promote products derived |
| from this software without specific prior written permission. |
| |
| THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS |
| "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT |
| LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR |
| A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL SYNTHETIC |
| SEMANTICS LLC BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, |
| EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, |
| PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR |
| PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF |
| LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING |
| NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS |
| SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. |
| |
+-----------------------------------------------------------------------------*/
#include "ems.h"
//==================================================================
// Push onto stack
int EMSpush(int mmapID, EMSvalueType *value) { // TODO: Eventually promote return value to 64bit
void *emsBuf = emsBufs[mmapID];
int64_t *bufInt64 = (int64_t *) emsBuf;
EMStag_t *bufTags = (EMStag_t *) emsBuf;
char *bufChar = (char *) emsBuf;
EMStag_t newTag;
// Wait until the stack top is full, then mark it busy while updating the stack
EMStransitionFEtag(&bufTags[EMScbTag(EMS_ARR_STACKTOP)], NULL, EMS_TAG_FULL, EMS_TAG_BUSY, EMS_TAG_ANY);
int32_t idx = bufInt64[EMScbData(EMS_ARR_STACKTOP)]; // TODO BUG: Truncating the full 64b range
bufInt64[EMScbData(EMS_ARR_STACKTOP)]++;
if (idx == bufInt64[EMScbData(EMS_ARR_NELEM)] - 1) {
fprintf(stderr, "EMSpush: Ran out of stack entries\n");
return -1;
}
// Wait until the target memory at the top of the stack is empty
newTag.byte = EMStransitionFEtag(&bufTags[EMSdataTag(idx)], NULL, EMS_TAG_EMPTY, EMS_TAG_BUSY, EMS_TAG_ANY);
newTag.tags.rw = 0;
newTag.tags.type = value->type;
newTag.tags.fe = EMS_TAG_FULL;
// Write the value onto the stack
switch (newTag.tags.type) {
case EMS_TYPE_BOOLEAN:
case EMS_TYPE_INTEGER:
case EMS_TYPE_FLOAT:
bufInt64[EMSdataData(idx)] = (int64_t) value->value;
break;
case EMS_TYPE_JSON:
case EMS_TYPE_STRING: {
int64_t textOffset;
EMS_ALLOC(textOffset, strlen((const char *) value->value) + 1, bufChar, "EMSpush: out of memory to store string\n", -1);
bufInt64[EMSdataData(idx)] = textOffset;
strcpy(EMSheapPtr(textOffset), (const char *) value->value);
}
break;
case EMS_TYPE_UNDEFINED:
bufInt64[EMSdataData(idx)] = 0xdeadbeef;
break;
default:
fprintf(stderr, "EMSpush: Unknown value type\n");
return -1;
}
// Mark the data on the stack as FULL
bufTags[EMSdataTag(idx)].byte = newTag.byte;
// Push is complete, Mark the stack pointer as full
bufTags[EMScbTag(EMS_ARR_STACKTOP)].tags.fe = EMS_TAG_FULL;
return idx;
}
//==================================================================
// Pop data from stack
//
bool EMSpop(int mmapID, EMSvalueType *returnValue) {
void *emsBuf = emsBufs[mmapID];
int64_t *bufInt64 = (int64_t *) emsBuf;
EMStag_t *bufTags = (EMStag_t *) emsBuf;
char *bufChar = (char *) emsBuf;
EMStag_t dataTag;
// Wait until the stack pointer is full and mark it empty while pop is performed
EMStransitionFEtag(&bufTags[EMScbTag(EMS_ARR_STACKTOP)], NULL, EMS_TAG_FULL, EMS_TAG_BUSY, EMS_TAG_ANY);
bufInt64[EMScbData(EMS_ARR_STACKTOP)]--;
int64_t idx = bufInt64[EMScbData(EMS_ARR_STACKTOP)];
if (idx < 0) {
// Stack is empty, return undefined
bufInt64[EMScbData(EMS_ARR_STACKTOP)] = 0;
bufTags[EMScbTag(EMS_ARR_STACKTOP)].tags.fe = EMS_TAG_FULL;
returnValue->type = EMS_TYPE_UNDEFINED;
returnValue->value = (void *) 0xf00dd00f;
return true;
}
// Wait until the data pointed to by the stack pointer is full, then mark it
// busy while it is copied, and set it to EMPTY when finished
dataTag.byte = EMStransitionFEtag(&bufTags[EMSdataTag(idx)], NULL, EMS_TAG_FULL, EMS_TAG_BUSY, EMS_TAG_ANY);
returnValue->type = dataTag.tags.type;
switch (dataTag.tags.type) {
case EMS_TYPE_BOOLEAN:
case EMS_TYPE_INTEGER:
case EMS_TYPE_FLOAT: {
returnValue->value = (void *) bufInt64[EMSdataData(idx)];
bufTags[EMSdataTag(idx)].tags.fe = EMS_TAG_EMPTY;
bufTags[EMScbTag(EMS_ARR_STACKTOP)].tags.fe = EMS_TAG_FULL;
return true;
}
case EMS_TYPE_JSON:
case EMS_TYPE_STRING: {
size_t memStrLen = strlen(EMSheapPtr(bufInt64[EMSdataData(idx)])); // TODO: Use size of allocation, not strlen
returnValue->value = malloc(memStrLen + 1); // freed in NodeJSfaa
if(returnValue->value == NULL) {
fprintf(stderr, "EMSpop: Unable to allocate space to return stack top string\n");
return false;
}
strcpy((char *) returnValue->value, EMSheapPtr(bufInt64[EMSdataData(idx)]));
EMS_FREE(bufInt64[EMSdataData(idx)]);
bufTags[EMSdataTag(idx)].tags.fe = EMS_TAG_EMPTY;
bufTags[EMScbTag(EMS_ARR_STACKTOP)].tags.fe = EMS_TAG_FULL;
return true;
}
case EMS_TYPE_UNDEFINED: {
bufTags[EMSdataTag(idx)].tags.fe = EMS_TAG_EMPTY;
bufTags[EMScbTag(EMS_ARR_STACKTOP)].tags.fe = EMS_TAG_FULL;
returnValue->value = (void *) 0xdeadbeef;
return true;
}
default:
fprintf(stderr, "EMSpop: ERROR - unknown top of stack data type\n");
return false;
}
}
//==================================================================
// Enqueue data
// Heap top and bottom are monotonically increasing, but the index
// returned is a circular buffer.
int EMSenqueue(int mmapID, EMSvalueType *value) { // TODO: Eventually promote return value to 64bit
void *emsBuf = emsBufs[mmapID];
int64_t *bufInt64 = (int64_t *) emsBuf;
EMStag_t *bufTags = (EMStag_t *) emsBuf;
char *bufChar = (char *) emsBuf;
// Wait until the heap top is full, and mark it busy while data is enqueued
EMStransitionFEtag(&bufTags[EMScbTag(EMS_ARR_STACKTOP)], NULL, EMS_TAG_FULL, EMS_TAG_BUSY, EMS_TAG_ANY);
int32_t idx = bufInt64[EMScbData(EMS_ARR_STACKTOP)] % bufInt64[EMScbData(EMS_ARR_NELEM)]; // TODO: BUG This could be truncated
bufInt64[EMScbData(EMS_ARR_STACKTOP)]++;
if (bufInt64[EMScbData(EMS_ARR_STACKTOP)] - bufInt64[EMScbData(EMS_ARR_Q_BOTTOM)] >
bufInt64[EMScbData(EMS_ARR_NELEM)]) {
fprintf(stderr, "EMSenqueue: Ran out of stack entries\n");
return -1;
}
// Wait for data pointed to by heap top to be empty, then set to Full while it is filled
bufTags[EMSdataTag(idx)].tags.rw = 0;
bufTags[EMSdataTag(idx)].tags.type = value->type;
switch (bufTags[EMSdataTag(idx)].tags.type) {
case EMS_TYPE_BOOLEAN:
case EMS_TYPE_INTEGER:
case EMS_TYPE_FLOAT:
bufInt64[EMSdataData(idx)] = (int64_t) value->value;
break;
case EMS_TYPE_JSON:
case EMS_TYPE_STRING: {
int64_t textOffset;
EMS_ALLOC(textOffset, strlen((const char *) value->value) + 1, bufChar, "EMSenqueue: out of memory to store string\n", -1);
bufInt64[EMSdataData(idx)] = textOffset;
strcpy(EMSheapPtr(textOffset), (const char *) value->value);
}
break;
case EMS_TYPE_UNDEFINED:
bufInt64[EMSdataData(idx)] = 0xdeadbeef;
break;
default:
fprintf(stderr, "EMSenqueue: Unknown value type\n");
return -1;
}
// Set the tag on the data to FULL
bufTags[EMSdataTag(idx)].tags.fe = EMS_TAG_FULL;
// Enqueue is complete, set the tag on the heap to to FULL
bufTags[EMScbTag(EMS_ARR_STACKTOP)].tags.fe = EMS_TAG_FULL;
return idx;
}
//==================================================================
// Dequeue
bool EMSdequeue(int mmapID, EMSvalueType *returnValue) {
void *emsBuf = emsBufs[mmapID];
int64_t *bufInt64 = (int64_t *) emsBuf;
EMStag_t *bufTags = (EMStag_t *) emsBuf;
char *bufChar = (char *) emsBuf;
EMStag_t dataTag;
// Wait for bottom of heap pointer to be full, and mark it busy while data is dequeued
EMStransitionFEtag(&bufTags[EMScbTag(EMS_ARR_Q_BOTTOM)], NULL, EMS_TAG_FULL, EMS_TAG_BUSY, EMS_TAG_ANY);
int64_t idx = bufInt64[EMScbData(EMS_ARR_Q_BOTTOM)] % bufInt64[EMScbData(EMS_ARR_NELEM)];
// If Queue is empty, return undefined
if (bufInt64[EMScbData(EMS_ARR_Q_BOTTOM)] >= bufInt64[EMScbData(EMS_ARR_STACKTOP)]) {
bufInt64[EMScbData(EMS_ARR_Q_BOTTOM)] = bufInt64[EMScbData(EMS_ARR_STACKTOP)];
bufTags[EMScbTag(EMS_ARR_Q_BOTTOM)].tags.fe = EMS_TAG_FULL;
returnValue->type = EMS_TYPE_UNDEFINED;
returnValue->value = (void *) 0xf00dd00f;
return true;
}
bufInt64[EMScbData(EMS_ARR_Q_BOTTOM)]++;
// Wait for the data pointed to by the bottom of the heap to be full,
// then mark busy while copying it, and finally set it to empty when done
dataTag.byte = EMStransitionFEtag(&bufTags[EMSdataTag(idx)], NULL, EMS_TAG_FULL, EMS_TAG_BUSY, EMS_TAG_ANY);
dataTag.tags.fe = EMS_TAG_EMPTY;
returnValue->type = dataTag.tags.type;
switch (dataTag.tags.type) {
case EMS_TYPE_BOOLEAN:
case EMS_TYPE_INTEGER:
case EMS_TYPE_FLOAT: {
returnValue->value = (void *) bufInt64[EMSdataData(idx)];
bufTags[EMSdataTag(idx)].byte = dataTag.byte;
bufTags[EMScbTag(EMS_ARR_Q_BOTTOM)].tags.fe = EMS_TAG_FULL;
return true;
}
case EMS_TYPE_JSON:
case EMS_TYPE_STRING: {
bufTags[EMSdataTag(idx)].byte = dataTag.byte;
bufTags[EMScbTag(EMS_ARR_Q_BOTTOM)].tags.fe = EMS_TAG_FULL;
size_t memStrLen = strlen(EMSheapPtr(bufInt64[EMSdataData(idx)])); // TODO: Use size of allocation, not strlen
returnValue->value = malloc(memStrLen + 1); // freed in NodeJSfaa
if(returnValue->value == NULL) {
fprintf(stderr, "EMSdequeue: Unable to allocate space to return queue head string\n");
return false;
}
strcpy((char *) returnValue->value, EMSheapPtr(bufInt64[EMSdataData(idx)]));
EMS_FREE(bufInt64[EMSdataData(idx)]);
return true;
}
case EMS_TYPE_UNDEFINED: {
bufTags[EMSdataTag(idx)].byte = dataTag.byte;
bufTags[EMScbTag(EMS_ARR_Q_BOTTOM)].tags.fe = EMS_TAG_FULL;
returnValue->value = (void *) 0xdeadbeef;
return true;
}
default:
fprintf(stderr, "EMSdequeue: ERROR - unknown type at head of queue\n");
return false;
}
}
================================================
FILE: src/rmw.cc
================================================
/*-----------------------------------------------------------------------------+
| Extended Memory Semantics (EMS) Version 1.3.0 |
| Synthetic Semantics http://www.synsem.com/ mogill@synsem.com |
+-----------------------------------------------------------------------------+
| Copyright (c) 2011-2014, Synthetic Semantics LLC. All rights reserved. |
| Copyright (c) 2015-2016, Jace A Mogill. All rights reserved. |
| |
| Redistribution and use in source and binary forms, with or without |
| modification, are permitted provided that the following conditions are met: |
| * Redistributions of source code must retain the above copyright |
| notice, this list of conditions and the following disclaimer. |
| * Redistributions in binary form must reproduce the above copyright |
| notice, this list of conditions and the following disclaimer in the |
| documentation and/or other materials provided with the distribution. |
| * Neither the name of the Synthetic Semantics nor the names of its |
| contributors may be used to endorse or promote products derived |
| from this software without specific prior written permission. |
| |
| THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS |
| "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT |
| LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR |
| A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL SYNTHETIC |
| SEMANTICS LLC BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, |
| EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, |
| PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR |
| PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF |
| LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING |
| NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS |
| SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. |
| |
+-----------------------------------------------------------------------------*/
#include "ems.h"
//==================================================================
// Fetch and Add Atomic Memory Operation
// Returns a+b where a is data in EMS memory and b is an argument
bool EMSfaa(int mmapID, EMSvalueType *key, EMSvalueType *value, EMSvalueType *returnValue) {
void *emsBuf = emsBufs[mmapID];
volatile EMStag_t *bufTags = (EMStag_t *) emsBuf;
int64_t idx = EMSwriteIndexMap(mmapID, key);
volatile int64_t *bufInt64 = (int64_t *) emsBuf;
volatile double *bufDouble = (double *) emsBuf;
char *bufChar = (char *) emsBuf;
EMStag_t oldTag;
if (idx < 0 || idx >= bufInt64[EMScbData(EMS_ARR_NELEM)]) {
fprintf(stderr, "EMSfaa: index out of bounds\n");
return false;
}
volatile EMStag_t *maptag;
if (EMSisMapped) { maptag = &bufTags[EMSmapTag(idx)]; }
else { maptag = NULL; }
// Wait until the data is FULL, mark it busy while FAA is performed
oldTag.byte = EMStransitionFEtag(&bufTags[EMSdataTag(idx)], maptag,
EMS_TAG_FULL, EMS_TAG_BUSY, EMS_TAG_ANY);
oldTag.tags.fe = EMS_TAG_FULL; // When written back, mark FULL
switch (oldTag.tags.type) {
case EMS_TYPE_BOOLEAN: { // Bool + _______
bool retBool = bufInt64[EMSdataData(idx)]; // Read original value in memory
returnValue->value = (void *) retBool;
returnValue->type = EMS_TYPE_BOOLEAN;
switch (value->type) {
case EMS_TYPE_INTEGER: // Bool + Int
bufInt64[EMSdataData(idx)] += (int64_t) value->value;
oldTag.tags.type = EMS_TYPE_INTEGER;
break;
case EMS_TYPE_FLOAT: { // Bool + Float
EMSulong_double alias;
alias.u64 = (uint64_t) value->value;
bufDouble[EMSdataData(idx)] =
(double) bufInt64[EMSdataData(idx)] + alias.d;
oldTag.tags.type = EMS_TYPE_FLOAT;
}
break;
case EMS_TYPE_UNDEFINED: // Bool + undefined
bufDouble[EMSdataData(idx)] = NAN;
oldTag.tags.type = EMS_TYPE_FLOAT;
break;
case EMS_TYPE_BOOLEAN: // Bool + Bool
bufInt64[EMSdataData(idx)] += (int64_t) value->value;
oldTag.tags.type = EMS_TYPE_INTEGER;
break;
case EMS_TYPE_STRING: { // Bool + string
int64_t textOffset;
EMS_ALLOC(textOffset, value->length + 1 + 5, // String length + Terminating null + 'false'
bufChar, "EMSfaa(bool+string): out of memory to store string\n", false);
sprintf(EMSheapPtr(textOffset), "%s%s",
bufInt64[EMSdataData(idx)] ? "true" : "false", (const char *) value->value);
bufInt64[EMSdataData(idx)] = textOffset;
oldTag.tags.type = EMS_TYPE_STRING;
}
break;
default:
fprintf(stderr, "EMSfaa: Data is BOOL, but FAA arg type is unknown\n");
return false;
}
// Write the new type and set the tag to Full, then return the original value
bufTags[EMSdataTag(idx)].byte = oldTag.byte;
if (EMSisMapped) bufTags[EMSmapTag(idx)].tags.fe = EMS_TAG_FULL;
return true;
} // End of: Bool + ___
case EMS_TYPE_INTEGER: {
int64_t retInt = bufInt64[EMSdataData(idx)]; // Read original value in memory
returnValue->type = EMS_TYPE_INTEGER;
returnValue->value = (void *) retInt;
switch (value->type) {
case EMS_TYPE_INTEGER: { // Int + int
int64_t memInt = bufInt64[EMSdataData(idx)] + (int64_t) value->value;
// TODO: Magic max int promotion to float
if (memInt >= (1 << 30)) { // Possible integer overflow, convert to float
bufDouble[EMSdataData(idx)] = (double) memInt;
oldTag.tags.type = EMS_TYPE_FLOAT;
} else { // Did not overflow to flow, still an integer
bufInt64[EMSdataData(idx)] = memInt;
}
}
break;
case EMS_TYPE_FLOAT: { // Int + float
EMSulong_double alias;
alias.u64 = (uint64_t) value->value;
bufDouble[EMSdataData(idx)] =
(double) bufInt64[EMSdataData(idx)] + alias.d;
oldTag.tags.type = EMS_TYPE_FLOAT;
}
break;
case EMS_TYPE_UNDEFINED: // Int + undefined
bufDouble[EMSdataData(idx)] = NAN;
oldTag.tags.type = EMS_TYPE_FLOAT;
break;
case EMS_TYPE_BOOLEAN: // Int + bool
bufInt64[EMSdataData(idx)] += (int64_t) value->value;
break;
case EMS_TYPE_STRING: { // int + string
int64_t textOffset;
EMS_ALLOC(textOffset, value->length + 1 + MAX_NUMBER2STR_LEN,
bufChar, "EMSfaa(int+string): out of memory to store string\n", false);
sprintf(EMSheapPtr(textOffset), "%lld%s",
(long long int) bufInt64[EMSdataData(idx)], (const char *) value->value);
bufInt64[EMSdataData(idx)] = textOffset;
oldTag.tags.type = EMS_TYPE_STRING;
}
break;
default:
fprintf(stderr, "EMSfaa: Data is INT, but FAA arg type is unknown\n");
return false;
}
// Write the new type and set the tag to Full, then return the original value
bufTags[EMSdataTag(idx)].byte = oldTag.byte;
if (EMSisMapped) bufTags[EMSmapTag(idx)].tags.fe = EMS_TAG_FULL;
return true;
} // End of: Integer + ____
case EMS_TYPE_FLOAT: {
double retDbl = bufDouble[EMSdataData(idx)];
returnValue->type = EMS_TYPE_FLOAT;
EMSulong_double alias;
alias.d = retDbl;
returnValue->value = (void *) alias.u64;
switch (value->type) {
case EMS_TYPE_INTEGER: // Float + int
bufDouble[EMSdataData(idx)] += (double) ((int64_t) value->value);
break;
case EMS_TYPE_FLOAT: { // Float + float
EMSulong_double alias;
alias.u64 = (uint64_t) value->value;
bufDouble[EMSdataData(idx)] += alias.d;
}
break;
case EMS_TYPE_BOOLEAN: // Float + boolean
bufDouble[EMSdataData(idx)] += (double) ((int64_t) value->value);
break;
case EMS_TYPE_STRING: { // Float + string
int64_t textOffset;
EMS_ALLOC(textOffset, value->length + 1 + MAX_NUMBER2STR_LEN,
bufChar, "EMSfaa(float+string): out of memory to store string\n", false);
sprintf(EMSheapPtr(textOffset), "%lf%s", bufDouble[EMSdataData(idx)], (const char *) value->value);
bufInt64[EMSdataData(idx)] = textOffset;
oldTag.tags.type = EMS_TYPE_STRING;
}
break;
case EMS_TYPE_UNDEFINED: // Float + Undefined
bufDouble[EMSdataData(idx)] = NAN;
break;
default:
fprintf(stderr, "EMSfaa: Data is FLOAT, but arg type unknown\n");
return false;
}
// Write the new type and set the tag to Full, then return the original value
bufTags[EMSdataTag(idx)].byte = oldTag.byte;
if (EMSisMapped) bufTags[EMSmapTag(idx)].tags.fe = EMS_TAG_FULL;
return true;
} // End of: float + _______
case EMS_TYPE_STRING: {
// size_t oldStrLen = (size_t) emsMem_size(EMS_MEM_MALLOCBOT(bufChar), bufInt64[EMSdataData(idx)]);
size_t oldStrLen = strlen(EMSheapPtr(bufInt64[EMSdataData(idx)]));
returnValue->type = EMS_TYPE_STRING;
returnValue->value = malloc(oldStrLen + 1); // freed in NodeJSfaa
if(returnValue->value == NULL) {
fprintf(stderr, "EMSfaa: Unable to malloc temporary old string\n");
return false;
}
strcpy((char *) returnValue->value, EMSheapPtr(bufInt64[EMSdataData(idx)]));
int64_t textOffset;
size_t len;
switch (value->type) {
case EMS_TYPE_INTEGER: // string + int
len = oldStrLen + 1 + MAX_NUMBER2STR_LEN;
EMS_ALLOC(textOffset, len, bufChar, "EMSfaa(string+int): out of memory to store string\n", false);
sprintf(EMSheapPtr(textOffset), "%s%lld",
EMSheapPtr(bufInt64[EMSdataData(idx)]),
(long long int) value->value);
break;
case EMS_TYPE_FLOAT: { // string + dbl
EMSulong_double alias;
alias.u64 = (uint64_t) value->value;
len = oldStrLen + 1 + MAX_NUMBER2STR_LEN;
EMS_ALLOC(textOffset, len, bufChar, "EMSfaa(string+dbl): out of memory to store string\n", false);
sprintf(EMSheapPtr(textOffset), "%s%lf",
EMSheapPtr(bufInt64[EMSdataData(idx)]), alias.d);
}
break;
case EMS_TYPE_STRING: { // string + string
len = oldStrLen + 1 + value->length;
EMS_ALLOC(textOffset, len, bufChar, "EMSfaa(string+string): out of memory to store string\n", false);
sprintf(EMSheapPtr(textOffset), "%s%s",
EMSheapPtr(bufInt64[EMSdataData(idx)]), (const char *) value->value);
}
break;
case EMS_TYPE_BOOLEAN: // string + bool
len = strlen(EMSheapPtr(bufInt64[EMSdataData(idx)])) + 1 + 5; // 5==strlen("false")
EMS_ALLOC(textOffset, len, bufChar, "EMSfaa(string+bool): out of memory to store string\n", false);
sprintf(EMSheapPtr(textOffset), "%s%s",
EMSheapPtr(bufInt64[EMSdataData(idx)]), (bool) value->value ? "true" : "false");
break;
case EMS_TYPE_UNDEFINED: // string + undefined
len = strlen(EMSheapPtr(bufInt64[EMSdataData(idx)])) + 1 + 9; // 9 == strlen("undefined");
EMS_ALLOC(textOffset, len, bufChar, "EMSfaa(string+undefined): out of memory to store string\n", false);
sprintf(EMSheapPtr(textOffset), "%s%s",
EMSheapPtr(bufInt64[EMSdataData(idx)]), "undefined");
break;
default:
fprintf(stderr, "EMSfaa(string+?): Unknown data type\n");
return false;
}
EMS_FREE(bufInt64[EMSdataData(idx)]);
bufInt64[EMSdataData(idx)] = textOffset;
oldTag.tags.type = EMS_TYPE_STRING;
// Write the new type and set the tag to Full, then return the original value
bufTags[EMSdataTag(idx)].byte = oldTag.byte;
if (EMSisMapped) bufTags[EMSmapTag(idx)].tags.fe = EMS_TAG_FULL;
// return value was set at the top of this block
return true;
} // End of: String + __________
case EMS_TYPE_UNDEFINED: {
returnValue->type = EMS_TYPE_UNDEFINED;
returnValue->value = (void *) 0xf00dd00f;
switch (value->type) { // Undefined + Int, dloat, bool, or undef
case EMS_TYPE_INTEGER:
case EMS_TYPE_FLOAT:
case EMS_TYPE_BOOLEAN:
case EMS_TYPE_UNDEFINED:
bufDouble[EMSdataData(idx)] = NAN;
oldTag.tags.type = EMS_TYPE_FLOAT;
break;
case EMS_TYPE_STRING: { // Undefined + string
int64_t textOffset;
EMS_ALLOC(textOffset, value->length + 1 + 3, // 3 = strlen("NaN");
bufChar, "EMSfaa(undef+String): out of memory to store string\n", false);
sprintf(EMSheapPtr(textOffset), "NaN%s", (const char *) value->value);
bufInt64[EMSdataData(idx)] = textOffset;
oldTag.tags.type = EMS_TYPE_UNDEFINED;
}
break;
default:
fprintf(stderr, "EMSfaa(Undefined+___: Unknown stored data type\n");
return false;
}
// Write the new type and set the tag to Full, then return the original value
bufTags[EMSdataTag(idx)].byte = oldTag.byte;
if (EMSisMapped) bufTags[EMSmapTag(idx)].tags.fe = EMS_TAG_FULL;
return true;
}
default:
fprintf(stderr, "EMSfaa(?+___: Unknown stored data type\n");
return false;
}
// fprintf(stderr, "EMSfaa: Unknown everything -- this statement is unreachable\n");
return false;
}
//==================================================================
// Atomic Compare and Swap
bool EMScas(int mmapID, EMSvalueType *key,
EMSvalueType *oldValue, EMSvalueType *newValue,
EMSvalueType *returnValue) {
void *emsBuf = emsBufs[mmapID];
volatile int64_t *bufInt64 = (int64_t *) emsBuf;
int64_t idx = EMSkey2index(emsBuf, key, EMSisMapped);
char * bufChar = (char *) emsBuf;
volatile EMStag_t *bufTags = (EMStag_t *) emsBuf;
EMStag_t newTag;
int64_t textOffset;
int swapped = false;
if ((!EMSisMapped && idx < 0) || idx >= bufInt64[EMScbData(EMS_ARR_NELEM)]) {
fprintf(stderr, "EMScas: index out of bounds\n");
return false;
}
size_t memStrLen;
unsigned char memType;
retry_on_undefined:
if(EMSisMapped && idx < 0) {
memType = EMS_TYPE_UNDEFINED;
} else {
// Wait for the memory to be Full, then mark it Busy while CAS works
volatile EMStag_t *maptag;
if (EMSisMapped) { maptag = &bufTags[EMSmapTag(idx)]; }
else { maptag = NULL; }
// Wait until the data is FULL, mark it busy while FAA is performed
EMStransitionFEtag(&bufTags[EMSdataTag(idx)], maptag,
EMS_TAG_FULL, EMS_TAG_BUSY, EMS_TAG_ANY);
memType = bufTags[EMSdataTag(idx)].tags.type;
}
// Read the value in memory
returnValue->type = memType;
switch (memType) {
case EMS_TYPE_UNDEFINED:
returnValue->value = (void *) 0xf00dcafe;
break;
case EMS_TYPE_BOOLEAN:
case EMS_TYPE_INTEGER:
case EMS_TYPE_FLOAT:
returnValue->value = (void *) bufInt64[EMSdataData(idx)];
break;
case EMS_TYPE_JSON:
case EMS_TYPE_STRING:
memStrLen = strlen(EMSheapPtr(bufInt64[EMSdataData(idx)]));
returnValue->value = malloc(memStrLen + 1); // freed in NodeJSfaa
if(returnValue->value == NULL) {
fprintf(stderr, "EMScas: Unable to allocate space to return old string\n");
return false;
}
strcpy((char *) returnValue->value, EMSheapPtr(bufInt64[EMSdataData(idx)]));
break;
default:
fprintf(stderr, "EMScas: memType not recognized\n");
return false;
}
// Compare the value in memory the the "old" CAS value
if (oldValue->type == memType) {
// Allocate on Write: If this memory was undefined (ie: unallocated),
// allocate the index map, store the undefined, and start over again.
if(EMSisMapped && idx < 0) {
idx = EMSwriteIndexMap(mmapID, key);
if (idx < 0) {
fprintf(stderr, "EMScas: Not able to allocate map on CAS of undefined data\n");
return false;
}
bufInt64[EMSdataData(idx)] = 0xcafebabe;
newTag.tags.fe = EMS_TAG_FULL;
newTag.tags.rw = 0;
newTag.tags.type = EMS_TYPE_UNDEFINED;
bufTags[EMSdataTag(idx)].byte = newTag.byte;
goto retry_on_undefined;
}
switch (memType) {
case EMS_TYPE_UNDEFINED:
swapped = true;
break;
case EMS_TYPE_BOOLEAN:
case EMS_TYPE_INTEGER:
case EMS_TYPE_FLOAT:
if (returnValue->value == oldValue->value)
swapped = true;
break;
case EMS_TYPE_JSON:
case EMS_TYPE_STRING:
if (strcmp((const char *) returnValue->value, (const char *) oldValue->value) == 0) {
swapped = true;
}
break;
default:
fprintf(stderr, "EMScas: oldTag not recognized");
return false;
}
}
// If memory==old then write the new value
newTag.tags.fe = EMS_TAG_FULL;
newTag.tags.rw = 0;
newTag.tags.type = memType;
if (swapped) {
if (memType == EMS_TYPE_STRING || memType == EMS_TYPE_JSON)
EMS_FREE((size_t) bufInt64[EMSdataData(idx)]);
newTag.tags.type = newValue->type;
switch (newValue->type) {
case EMS_TYPE_UNDEFINED:
bufInt64[EMSdataData(idx)] = 0xbeeff00d;
break;
case EMS_TYPE_BOOLEAN:
case EMS_TYPE_INTEGER:
case EMS_TYPE_FLOAT:
bufInt64[EMSdataData(idx)] = (int64_t) newValue->value;
break;
case EMS_TYPE_JSON:
case EMS_TYPE_STRING:
EMS_ALLOC(textOffset, newValue->length + 1,
bufChar, "EMScas(string): out of memory to store string\n", false);
strcpy(EMSheapPtr(textOffset), (const char *) newValue->value);
bufInt64[EMSdataData(idx)] = textOffset;
break;
default:
fprintf(stderr, "EMScas(): Unrecognized new type\n");
return false;
}
}
// Set the tag back to Full and return the original value
bufTags[EMSdataTag(idx)].byte = newTag.byte;
// If there is a map, set the map's tag back to full
if (EMSisMapped)
bufTags[EMSmapTag(idx)].tags.fe = EMS_TAG_FULL;
return true;
}
fork primitives,
creates the additional threads executing
using one of two execution models: fork-join or Bulk Synchronous Parallel (BSP).
BSP invokes the same script as the master thread (found in 
================================================
FILE: Examples/Interlanguage/interlanguage.js
================================================
/*-----------------------------------------------------------------------------+
| Extended Memory Semantics (EMS) Version 1.4.5 |
| Synthetic Semantics http://www.synsem.com/ mogill@synsem.com |
+-----------------------------------------------------------------------------+
| Copyright (c) 2017, Jace A Mogill. All rights reserved. |
| |
| Redistribution and use in source and binary forms, with or without |
| modification, are permitted provided that the following conditions are met: |
| * Redistributions of source code must retain the above copyright |
| notice, this list of conditions and the following disclaimer. |
| * Redistributions in binary form must reproduce the above copyright |
| notice, this list of conditions and the following disclaimer in the |
| documentation and/or other materials provided with the distribution. |
| * Neither the name of the Synthetic Semantics nor the names of its |
| contributors may be used to endorse or promote products derived |
| from this software without specific prior written permission. |
| |
| THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS |
| "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT |
| LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR |
| A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL SYNTHETIC |
| SEMANTICS LLC BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, |
| EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, |
| PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR |
| PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF |
| LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING |
| NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS |
| SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. |
| |
+-----------------------------------------------------------------------------*/
/*
Start this JS program first, then start the Python program
Possible Orders for Starting Processes
---------------------------------------------
A persistent EMS file already exists
1. A one-time initialization process creates the EMS array
2. Any number of processes my attach to the EMS array in
any order.
A new EMS array will be created
1. The first program to start must create the EMS file with
the "useExisting : false" attribute
2. Subsequent programs attach to the new EMS file with the
"useExisting : true" attribute
*/
"use strict";
// Initialize EMS with 1 process, no thread-CPU affinity,
// "user" mode parallelism, and a unique namespace for EMS runtime
// ("jsExample") to keep the JS program distinct from Python EMS
// program also running.
let ems = require("ems")(1, false, "user", "jsExample");
if (Number(process.version[1]) < 6) {
console.log('Requires Node.js version 6.x or higher -- will not work with polyfill Harmony');
process.exit(1);
}
const maxNKeys = 100;
const bytesPerEntry = 100; // Bytes of storage per key, used for key (dictionary word) itself
let shared = ems.new({
"ES6proxies": true, // Enable native JS object-like syntax
"useExisting": false, // Create a new EMS memory area, do not use existing one if present
"filename": "/tmp/interlanguage.ems", // Persistent EMS array's filename
"dimensions": maxNKeys, // Maximum # of different keys the array can store
"heapSize": maxNKeys * bytesPerEntry,
"setFEtags": "empty", // Set default full/empty state of EMS memory to empty
"doSetFEtags": true,
"useMap": true // Use a key-index mapping, not integer indexes
});
//------------------------------------------------------------------------------------------
// Begin Main Program
console.log("JS Started, waiting for Python to start...");
// One-time initialization should be performed before syncing to Py
shared.writeXF("nestedObj", undefined);
// Initial synchronization with Python
// Write a value to empty memory and mark it full
shared.writeEF("JS hello", "from Javascript");
// Wait for Python to start
console.log("Hello " + shared.readFE("Py hello"));
/*
This synchronous exchange was a "barrier" between the two processes.
The idiom of exchanging messages by writeEF and readFE constitutes
a synchronization point between two processes. A barrier synchronizing
N tasks would require N² variables, which is a reasonable way to
implement a barrier when N is small. For larger numbers of processes
barriers can be implemented using shared counters and the Fetch-And-Add
(FAA) EMS instruction.
The initialization of EMS values as "empty" occurs when the EMS array
was created with:
"setFEtags": "empty",
"doSetFEtags": true,
If it becomes necessary to reset a barrier, writeXE() will
unconditionally and immediately write the value and mark it empty.
*/
// Convenience function to synchronize with another process
// The Python side reverses the barrier names
function barrier(message) {
console.log("Entering Barrier:", message);
shared.writeEF("py side barrier", undefined);
shared.readFE("js side barrier");
console.log("Completed Barrier.");
console.log("---------------------------------------------------");
}
// --------------------------------------------------------------------
barrier("Trying out the barrier utility function");
// --------------------------------------------------------------------
// JS and Python EMS can read sub-objects with the same syntax as native objects,
// but writes are limited to only top level attributes. This corresponds to the
// implied "emsArray.read(key).subobj" performed
shared.top = {};
console.log("Assignment to top-level attributes works normally:", shared.top);
shared.top.subobj = 1; // Does not take effect like top-level attributes
console.log("But sub-object attributes do not take effect. No foo?", shared.top.subobj);
// A nested object can be written at the top level.
shared.nestedObj = {"one":1, "subobj":{"left":"l", "right":"r"}};
console.log("Nested object read references work normally", shared.nestedObj.subobj.left);
// A workaround to operating on nested objects
let nestedObjTemp = shared.nestedObj;
nestedObjTemp.subobj.middle = "m";
shared.nestedObj = nestedObjTemp;
// The explicitly parallel-safe way to modify objects is similar to that "workaround"
nestedObjTemp = shared.readFE("nestedObj");
nestedObjTemp.subobj.front = "f";
shared.writeEF("nestedObj", nestedObjTemp);
// --------------------------------------------------------------------
barrier("This barrier matches where Python is waiting to use nestedObj");
// --------------------------------------------------------------------
// Python does some work now
// Initialize the counter for the next section
shared.writeXF("counter", 0);
// --------------------------------------------------------------------
barrier("JS and Py are synchronized at the end of working with nestedObj");
// --------------------------------------------------------------------
// ---------------------------------------------------------------------------
// Use the EMS atomic operation intrinsics to coordinate access to data
// This is 10x as many iterations as Python in the same amount of time
// because V8's TurboFan compiler kicks in.
for (let count = 0; count < 1000000; count += 1) {
let value = shared.faa("counter", 1);
if (count % 100000 == 0) {
console.log("JS iteration", count, " Shared Counter=", value)
}
}
// --------------------------------------------------------------------
barrier("Waiting for Py to finish it's counter loop");
// --------------------------------------------------------------------
console.log("The shared counter should be 11000000 ==", shared.counter);
// ---------------------------------------------------------------------------
// Ready to earn your Pro Card?
// Wait for Py to initialize the array+string
barrier("Wait until it's time to glimpse into the future of Javascript");
console.log("The array+string from Py:", shared.arrayPlusString);
barrier("Waiting for shared.arrayPlusString to be initialized");
console.log("JS can do array+string, it produces a string:", shared.arrayPlusString + " world!");
shared.arrayPlusString += " RMW world!";
console.log("JS performing \"array + string\" produces a self-consistent string:", shared.arrayPlusString);
barrier("Arrive at the future.");
// ---------------------------------------------------------------------------
// Leave a counter running on a timer
// First re-initialize the counter to 0
shared.writeXF("counter", 0);
function incrementCounter() {
// Atomically increment the shared counter
let oldValue = shared.faa("counter", 1);
// If the counter has been set to "Stop", clear the timer interval
if (oldValue == "stop") {
console.log("Timer counter was signaled to stop");
clearInterval(timerCounter);
}
// Show counter periodically
if (oldValue % 100 == 0) {
console.log("Counter =", shared.counter);
}
}
var timerCounter = setInterval(incrementCounter, 10); // Increment the counter every 1ms
barrier("Welcome to The Future™!");
// Fall into Node.js event loop with Interval Timer running
================================================
FILE: Examples/Interlanguage/interlanguage.py
================================================
#!/usr/bin/env python
# -*- coding: utf-8 -*-
"""
+-----------------------------------------------------------------------------+
| Extended Memory Semantics (EMS) Version 1.4.5 |
| Synthetic Semantics http://www.synsem.com/ mogill@synsem.com |
+-----------------------------------------------------------------------------+
| Copyright (c) 2017, Jace A Mogill. All rights reserved. |
| |
| Redistribution and use in source and binary forms, with or without |
| modification, are permitted provided that the following conditions are met: |
| * Redistributions of source code must retain the above copyright |
| notice, this list of conditions and the following disclaimer. |
| * Redistributions in binary form must reproduce the above copyright |
| notice, this list of conditions and the following disclaimer in the |
| documentation and/or other materials provided with the distribution. |
| * Neither the name of the Synthetic Semantics nor the names of its |
| contributors may be used to endorse or promote products derived |
| from this software without specific prior written permission. |
| |
| THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS |
| "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT |
| LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR |
| A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL SYNTHETIC |
| SEMANTICS LLC BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, |
| EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, |
| PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR |
| PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF |
| LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING |
| NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS |
| SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. |
| |
+-----------------------------------------------------------------------------+
"""
"""
tl;dr Start the Javascript first, then the Python
--------------------------------------------------
See interlanguage.js for more detailed description of starting processes
sharing EMS memory
NOTE ABOUT "STUCK" JOBS":
Python runtime can get "stuck" and may not respond to Control-C
but can still be interrupted with Control-Z or Control-\
This "stuck" behavior is due to how the Python/C interface protects
the Python runtime.
"""
import time
import sys
import random
sys.path.append("../../Python/")
import ems
# Initialize EMS: 1 process, no thread-core affinity, user provided parallelism
# Initialize EMS with 1 process, no thread-CPU affinity,
# "user" mode parallelism, and a unique namespace for EMS runtime
# ("pyExample") to keep the JS program distinct from Javascript EMS
# program also running.
ems.initialize(1, False, "user", "pyExample")
# The EMS array attributes must be the same by all programs sharing the array,
# with the exception of the the initial file creation which uses "useExisting:False"
# instead of "useExisting:True"
maxNKeys = 100
bytesPerEntry = 100 # Bytes of storage per key, used for key (dictionary word) itself
shared = ems.new({
'useExisting': True, # Connect the EMS memory created by JS, do not create a new memory
'filename': "/tmp/interlanguage.ems", # Persistent EMS array's filename
'dimensions': maxNKeys, # Maximum # of different keys the array can store
'heapSize': maxNKeys * 100, # 100 bytes of storage per key, used for key (dictionary word) itself
'useMap': True # Use a key-index mapping, not integer indexes
})
# ------------------------------------------------------------------------------------------
# Begin Main Program
print """Start this program after the JS program.
If this Python appears hung and does not respond to ^C, also try ^\ and ^Z
----------------------------------------------------------------------------
"""
# Initial synchronization with JS
# Write a value to empty memory and mark it full
shared.writeEF("Py hello", "from Python")
# Wait for JS to start
print "Hello ", shared.readFE("JS hello")
def barrier(message):
"""Define a convenience function to synchronize with another process
The JS side reverses the barrier names"""
global shared
naptime = random.random() + 0.5
time.sleep(naptime) # Delay 0.5-1.5sec to make synchronization apparent
print "Entering Barrier:", message
shared.writeEF("js side barrier", None)
shared.readFE("py side barrier")
print "Completed Barrier after delaying for", naptime, "seconds"
print "------------------------------------------------------------------"
print "Trying out the new barrier by first napping for 1 second..."
time.sleep(1)
# ---------------------------------------------------------------------
barrier("Trying out the barrier utility function")
# ---------------------------------------------------------------------
# --------------------------------------------------------------------
# This barrier matches where JS has set up "shared.top", and is now waiting
# for Python to finish at the next barrier
barrier("Finished waiting for JS to finish initializing nestedObj")
print "Value of shared.nestedObj left by JS:", shared.nestedObj
try:
shared.nestedObj.number = 987 # Like JS, setting attributes of sub-objects will not work
print "ERROR -- This should not work"
exit()
except:
print "Setting attributes of sub-objects via native syntax will not work"
print "These two expressions are the same:", \
shared["nestedObj"].read()["subobj"]["middle"], "or", \
shared.read("nestedObj")["subobj"]["middle"]
# --------------------------------------------------------------------
barrier("JS and Py are synchronized at the end of working with nestedObj")
# --------------------------------------------------------------------
# Enter the shared counter loop
for count in xrange(100000):
value = shared.faa("counter", 1)
if count % 10000 == 0:
print "Py iteration", count, " Shared counter=", value
barrier("Waiting for Js to finish it's counter loop")
# --------------------------------------------------------------------
# Ready to earn your Pro Card?
# Make JS wait while Python demonstrates a "create approach to language design"
shared.arrayPlusString = [1, 2, 3] # Initialize arrayPlusString before starting next phase
barrier("Wait until it's time to glimpse into the future of Python")
try:
dummy = shared.arrayPlusString + "hello"
print "ERROR -- Should result in: TypeError: can only concatenate list (not \"str\") to list"
exit()
except:
print "Adding an array and string is an error in Python"
shared.arrayPlusString += "hello"
print "However 'array += string' produces an array", shared.arrayPlusString
# Let JS know 'shared.arrayPlusString' was initialized
barrier("shared.arrayPlusString is now initialized")
# JS does it's work now, then wait for JS to do it's work
barrier("Arrive at the future.")
# JS does all the work in this section
barrier("Welcome to The Future™!")
# --------------------------------------------------------------------
# JS is now in it's event loop with the counter running
for sampleN in xrange(10):
time.sleep(random.random())
print "Shared counter is now", shared.counter
# Restart the counter
shared.counter = 0
for sampleN in xrange(5):
time.sleep(random.random() + 0.5)
print "Shared reset counter is now", shared.counter
# Wait 1 second and stop the JS timer counter
time.sleep(1)
shared.counter = "stop"
# Show the counter has stopped. Last value = "stop" + 1
for sampleN in xrange(5):
time.sleep(random.random() + 0.5)
print "Shared reset counter should have stopped changing:", shared.counter
print "Exiting normally."
================================================
FILE: Examples/KeyValueStore/README.md
================================================
# Key-Value Store Example
This example implements a simple REST server, the API is a 1-to-1
mapping of REST commands to EMS commands,
effectively making the EMS shared memory model
an object store service over the network.
The RESTful API inherits all the atomicity
and persistence properties of the underlying EMS array.
## Parallelism & Deadlocks
The HTTP web interface used is Node's built-in HTTP module which
implements a serial event loop to process HTTP requests one at
time in a synchronous (blocking) fashion.
The execution time of EMS operations is similar to any other variable reference,
making EMS most efficient as a synchronous programming model,
however, because a `readFE` request may block the Node event loop
clients susceptible to deadlock.
A `writeEF` request that would unblock execution cannot handled
by the single-threaded Node HTTP server that is blocked on a
`readFE`, effectively deadlocking the system.
The two potential solutions, asynchronous execution and multi-processing,
are complementary forms of parallelism that may be used
to address the deadlock problem:
- __Asynchronous Execution__:
EMS is a memory model and has no daemon process to monitor a network interface
for operations to execute.
This example builds on a standard HTTP server to implement
shared memory parallelism within a single web request.
## EMS as a Key Value Store
Click here for the [EMS Key-Value Store Example](https://github.com/SyntheticSemantics/ems/tree/master/Examples/KeyValueStore)
An EMS array is used as a Key-Value store with a 1-to-1 mapping of REST operations
to EMS memory operations.
The KV server is implemented using the Node.js built-in `Cluster` module to
demonstrate parallelism sources outside of EMS.
## Work Queues
The parameters of the experiment are defined in the `initializeSharedData()` function.
An experiment creates several new EMS arrays and a work queue of
randomly generated transactions that update one or more of the arrays.
Parallel processes dequeue the work and perform atomic updates on
the new EMS arrays. Finally, the experiment performs checksums to ensure
all the transactions were performed properly.
```javascript
arrLen = 1000000; // Maximum number of elements the EMS array can contain
heapSize = 100000; // Amount of persistent memory to reserve for transactions
nTransactions = 1000000; // Number of transactions to perform in the experiment
nTables = 6; // Number of EMS arrays to perform transactions across
maxNops = 5; // Maximum number of EMS arrays to update during a transaction
```
### `concurrent_Q_and_TM.js`
```bash
node concurrent_Q_and_TM.js 4 # 1 process enqueues then processes work, 3 processes perform work
````
All processes are started, one enqueues transactions while the others dequeue
and perform the work. When all the work has been enqueued, that process also
begins dequeing and performing work.
### `workQ_and_TM.js`
```bash
node workQ_and_TM.js 4 # Enqueue all the work, all 4 processes deqeue and perform work
````
All the transactions are enqueued, then all the processes begin
to dequeue and perform work until no work remains in the queue.
## [Word Counting](#Word-Count)
Map-Reduce is often demonstrated using word counting because each document can
be processed in parallel, and the results of each document's dictionary reduced
into a single dictionary.
In contrast with Map-Reduce implementations using a
dictionary per document, the EMS implementation
maintains a single shared dictionary, eliminating the need for a reduce phase.
The word counts are sorted in parallel and the most frequently appearing words
are printed with their counts.
### Word Counting Using Atomic Operations
A parallel word counting program is implemented in Python and Javascript.
A shared dictionary is used by all the tasks, updates are made using
atomic operations. Specifically, `Compare And Swap (CAS)` is used to
initialize a new word from undefined to a count of 1. If this CAS
fails, the word already exists in the dictionary and the operation
is retried using `Fetch and Add`, atomically incrementing the count.
### Forming the "Bag of Words" with Word Counts
```javascript
var file_list = fs.readdirSync(doc_path);
var splitPattern = new RegExp(/[ \n,\.\\/_\-\<\>:\;\!\@\#\$\%\&\*\(\)=\[\]|\"\'\{\}\?\—]/);
// Iterate over all the files in parallel
ems.parForEach(0, file_list.length, function (fileNum) {
var text = fs.readFileSync(doc_path + file_list[fileNum], 'utf8', "r");
var words = text.replace(/[\n\r]/g, ' ').toLowerCase().split(splitPattern);
// Sequentially iterate over all the words in one document
words.forEach(function (word) {
wordCounts.faa(word, 1); // Atomically increment the count of times this word was seen
});
});
```
### Sorting the Word Count list by Frequency
Parallel sorting of the word count is implemented as a multi-step process:
1. The bag of words is processed by all the processess, each process
building an ordered list of the most common words it encounters
2. The partial lists from all the processes are concatenated into a single list
3. The list of the most common words seen is sorted and truncated
```javascript
var biggest_counts = new Array(local_sort_len).fill({"key": 0, "count": 0});
ems.parForEach(0, maxNKeys, function (keyN) {
var key = wordCounts.index2key(keyN);
if (key) {
// Perform an insertion sort of the new key into the biggest_counts
// list, deleting the last (smallest) element to preserve length.
var keyCount = wordCounts.read(key);
var idx = local_sort_len - 1;
while (idx >= 0 && biggest_counts[idx].count < keyCount) { idx -= 1; }
var newBiggest = {"key": key, "count": keyCount};
if (idx < 0) {
biggest_counts = [newBiggest].concat(biggest_counts.slice(0, biggest_counts.length - 1));
} else if (idx < local_sort_len) {
var left = biggest_counts.slice(0, idx + 1);
var right = biggest_counts.slice(idx + 1);
biggest_counts = left.concat([newBiggest].concat(right)).slice(0, -1);
}
}
});
// Concatenate all the partial (one per process) lists into one list
stats.writeXF('most_frequent', []); // Initialize the list
ems.barrier(); // Wait for all procs to have finished initialization
stats.writeEF('most_frequent', stats.readFE('most_frequent').concat(biggest_counts));
ems.barrier(); // Wait for all procs to have finished merge
ems.master(function() { // Sort & print the list of words, only one process is needed
biggest_counts = stats.readFF('most_frequent');
biggest_counts.sort(function (a, b) { return b.count - a.count; });
// Print only the first "local_sort_len" items -- assume the worst case
// of all the largest counts are discovered by a single process
console.log("Most frequently appearing terms:");
for (var index = 0; index < local_sort_len; index += 1) {
console.log(index + ': ' + biggest_counts[index].key + " " + biggest_counts[index].count);
}
});
```
### Word Count Performance
This section reports the results of running the Word Count example on
documents from Project Gutenberg.
2,981,712,952 words in several languages were parsed, totaling 12,664,852,220 bytes of text.
## Inter-Language Programming
[Inter-Language Shared Memory Programming](https://github.com/SyntheticSemantics/ems/tree/master/Examples/Interlanguage)
The programs `interlanguage.js` and `interlanguage.py` demonstrate sharing
objects between Javascript and Python.
A variety of synchronization and execution models are shown.
### Example
###### Copyright (C)2011-2019 Jace A Mogill
================================================
FILE: Examples/STREAMS/README.md
================================================
# EMS.js STREAMS Bandwidth Benchmark
The original [STREAMS](https://www.cs.virginia.edu/stream/)
benchmark measures the _Speed Not To Exceed_
bandwidth of a CPU to it's attached RAM by performing large
numbers of simple operations. This version for
EMS
## REST API
A valid request URL consists of path naming the session ID
and a query that specifies if the session is new/restarting, or
is a continuation of a previous session.
__REST API:__ `/sessionID?
Inter-language example in [interlanguage.{js,py}](https://github.com/mogill/ems/tree/master/Examples/Interlanguage)
The animated GIF demonstrates the following steps:
* Start Node.js REPL, create an EMS memory
* Store "Hello"
* Open a second session, begin the Python REPL
* Connect Python to the EMS shared memory
* Show the object created by JS is present in Python
* Modify the object, and show the modification can be seen in JS
* Exit both REPLs so no programs are running to "own" the EMS memory
* Restart Python, show the memory is still present
* Initialize a counter from Python
* Demonstrate atomic Fetch and Add in JS
* Start a loop in Python incrementing the counter
* Simultaneously print and modify the value from JS
* Try to read "empty" data from Python, the process blocks
* Write the empty memory, marking it full, Python resumes execution
## Types of Concurrency
The performance of this program was measured using an Amazon EC2 instance:
### Benchmarking of Transactions and Work Queues
[Transactions and Work Queues Example](https://github.com/mogill/ems/tree/master/Examples)
Transactional performance is measured alone, and again with a separate
process appending new processes as work is removed from the queue.
The experiments were run using an Amazon EC2 instance:
