The Problem
MapReduce (Hadoop)
void map(KEYIN key, VALUEIN value, Mapper.Context context)
void reduce(KEYIN key, Iterable values, Reducer.Context context)
- The domain must fit the MapReduce paradigm
- To easily scale later, MapReduce must be an early design choice
- The void return types imply reliance on side effects
The Holy Grail
- Initially build our application cheaply and quickly
- Later scale without changing the software
Follow the Data
- Let Swarm worry about where in the cluster data is located
- Use Swarm's Ref class to dereference the data
- Transparently relocate execution to the node with the data
Swarm
val local = new InetLocation(java.net.InetAddress.getLocalHost, 9998)
val remote = new InetLocation(java.net.InetAddress.getLocalHost, remotePort)
val a = Ref(local, "bumble bee")
val b = Ref(local, "honey bee")
val c = Ref(remote, "stingless bee")
println(a())
println(b())
println(c())
Delimited Continuations
object AsyncConsoleIO extends App {
type cont = cpsParam[Unit, Unit]
reset {
var i = 5
while (i > 0) {
shiftIt()
i -= 1
}
}
def shiftIt() = shift { k: (Unit => Unit) =>
val name = prompt()
k()
greet(name)
}
def prompt() = { println("name> "); readLine }
def greet(s: String) = { Thread.sleep(1000); println("hello, " + s) }
}
Delimited Continuations
def workflow(): NodeSeq @imp = {
val name = getName()
val age = getAge()
Hello {name}, you are {age} years old!
}
def getName(input: Option[String] = None): String @imp =
input match {
case None => getName(prompt(form("Name")))
case Some(name) => name
}
def getAge(input: Option[String] = None): String @imp =
input match {
case None => getAge(prompt(form("Age")))
case Some(age) if age.matches("\\d+") => age
case _ => getAge(prompt(
form("Age", Some("Your age must be a number."))))
}
Status of Swarm
Lots of cool stuff to do
- Data migration algorithm
- Reference relocation cleanup
- Cluster management
- Security
