blob: 23c88376666b41fa8db0de6c7b20f525b59b6dd3 [file] [log] [blame]
Use cases:
1. number of steps between 2 people in a graph (topology with cycles?)
#################
* Repackage jzmq and zmq as a leiningen "native dep"
- this might be good, since the native dep can package builds for all different systems/os's?
* Deploy design:
- storm swap {name} {jar} {class}
- it's allowed to use resources equal to current running topology plus number of free resources
- starts in deactivated mode
- add TOPOLOGY_STARTUP_TIME config for the delay until nimbus activates a topology after launching it
- for swap, after the startup time, deactivate the other topology, wait the TOPOLOGY_MESSAGE_TIMEOUT_SECS, and then activate the other topology
- should be able to decrease the message timeout for killing or swapping (add optional thrift parameter) -- or just make it part of the config?
- add killWithOptions, swap, swapWithOptions
* Storm UI, stats, debugging, diagnosis tools
-- need to be able to hide system streams/components from the calculations (another query param and should be default)
-- need to optimize (slowness is probably on nimbus end of querying zk, consider adding heartbeat caching into nimbus)
-- add margins
-- add titles so its easier to distinguish the various pages
-- right align all table columns except for the leftmost
* Unit test the core pieces that have stabilized their APIs
- process simulator
- virtual ports
- supervisor
- utils
- test worker/tasks
* implement pseudo-distributed mode -- this is for testing the distributed parts of the code
- perhaps i can use pallet/vmfest for this
* Need integration tests that run on an actual storm cluster (scp code/process code/zookeeper code not tested in unit tests)
* bolts with none grouping can be pushed into a bolt. e.g. A -> B -> C
A -> D -> E
If A -> B and A -> D are shuffle grouping = none, and B -> C and D -> E are not, then both can be run in A, b's branch goes to C and D's branch goes to E
* Failure design
Add fail method to outputcollector
Fail sends fail message to Acker for those anchors, which sends fail message back to spout.
Whenever spout fails a tuple, it emits it in its failure stream...
Add fail method to drpc... Causes blocked thread to throw exception
* Have worker heartbeat with its task ids, nimbus verifies - if wrong, reassign tasks?
- detect and ignore stray tasks
Each worker can choose a unique id for itself when heart beating
- nimbus deletes those that aren't in topology
* Subscriptions design
-- new kind of spout: "subscription spout"
--> goal is to sync it's data across the tasks that subscribe to its streams
--> after doing a grouping, remembers what task it sent the tuple to (regardless of grouping). if a task dies, it knows its subscriptions and asks to be resynced
--> normal operation is to push to tasks, but pull done when a task starts up (b/c previous task died or something)
--> need to be able to add tuples to subscription or take tuples away (this is protocol with who you're subscribing to - e.g. rocket)
--> subscriptions can only happen in a spout because it requires persistent state
--> when subscription spout task dies, it polls the source (e.g. rocket) for all the subscription info
--> ideally you'd set things up to have one subscription spout per rocket server
--> TODO: Need some way to delete subscriptions -> part of tuple or extra metadata on tuple (extra metadata seems cleaner)
--> add isSubscription() method to Tuple as well as a getSubscriptionType() [which returns ADD or REMOVE]
--> when a spout starts up, it also needs to push all of its subscription info
--> acks are irrelevant for subscription tuples -- how should acks be managed as an abstraction?
-- maybe the synchronized state is done for you -- you just access the state directly and receive a callback whenever it changes?
-- so don't use tuples...
--> subscriptions break all the abstractions, perhaps I should generalize spouts and factor acking as a library on top of storm. subscriptions would just be another kind of library? -> no, it seems to break abstractions anyway (like keeping task -> tuples in memory)
--> maybe call it "syncspout"
--> if just do syncing (don't expose tuples directly?)
--> have a "SubscribedState" class that takes care of indexing/etc. --> expose it through topologycontext?
-- need a way to distinguish between states of different streams
-- has "add" and "remove" methods
-- bolt can give a statemanager object that implements add and remove in the prepare method
-- add(Tuple tuple)
-- remove(Tuple tuple)
--> synchronize protocol (when spout or source of data dies):
--> send how many tuples are going to be sent
--> send the tuples
--> OR: pack everything together into a single message (could be hard b/c where tuples are supposed to go is abstracted away)
--> tie everything together with a unique ID
--> once task receives everything, has info needed to remove tuples
--> statespout should do long-polling with timeout
--> to do subscriptions, the state should contain something like [url, subscriber]. some bolt appends subscriber to tuples, group by subscriber, and send info back
--> how to to fields grouping with an even distribution?
--> ********* tasks need to block on startup until they're synchronized *********
--> send sync messages in a loop until it's synchronized
--> add a task.synchronize.poll.freq.secs config (default to 10 seconds)
--> need to buffer other messages as topology is waiting for synchronization messages (use disk?)
--> could use acking system to know if a piece of state gets fully synchronized and communicate this with user
--> perhaps expose this through a special stream? (the state status stream -> similar to failure streams)
--> should be able to do updates of existing state
--> use case: have a knob that you can set externally
--> this isn't really any better than just using zookeeper directly
_myState = context.setSubscribedState(_myState)
StateSpout {
//does a timeout long poll and emits new add or remove state tuples (add and remove on the output collector)
nextTuple(StateSpoutOutputCollector) //collector has add and remove methods add(id, tuple). remove(id)
//emits all the tuples into the output collector (in the background, will also send ids and counts to tasks so they know how to synchronize)
//called on startup
//collector can have a synchronize method in case the source of data (e.g., rocket) craps out
synchronize(SynchronizationOutputCollector) //collector only has add(id, tuple) method
}
//task startup (in prepare method) [this is automatic]
for(int taskId: statespoutids) {
emitDirect(SYNC_STREAM, tuple())
}
statespout synchronization():
id = uuid()
//getAlLStateTuples calls synchronize on the spout to get the tuples
for(Tuple t: getAllStateTuplesFromSource()) {
List tasks = emit(cons(id, t));
.. keep track of id -> tasks -> count
for(task: all output tasks) {
emitDirect(task, id, count)
}
}
for synchronization to work, task needs to keep track of which tasks sent it tuples, and compare against only that set on synchronization
Need a way to propogate information back up the topology - "subscriptions"
e.g. browser -> rocket -> bolt -> bolt -> bolt.
example: #retweets for a subscribed set of tweet ids
storm topology
-> tweet spout (A) -> group on original id -> count (B) -> rocket
subscriptions: rocket -> count (B) tweet id (need to group) -> spout (need to go to all)
-- how does it work when stuff dies downstream or upstream? do people ask what the subscriptions are? or do you push your subscriptions up? a combination?
-- maybe subscriptions are a "constant" spout? e..g, continuously emits and refreshes to make sure every task has the tuple. this seem amporphous and hard to implement... nimbus would need to refire all constant spouts whenever there's a reassignment that affects the flow of data. subscriptions seem more natural
-- subscriptions are a special kind of stream that are driven by being asked to send it. e..g, rocket is a spout that emits subscription/unsubscription tuples. they only send it when they get something new, or are asked as to what all the subscriptions are
-- maybe you just need a system stream to know when tasks are created. when you see that a downstream task is created, you know to fire subscriptions to it if its subscribed to your subscriptions stream? - how does this interplay with all the grouping types... you almost want to do a grouping and only send what to tasks that would have received. spouts would need to be able to subscribe to streams as well
(use 'backtype.storm.testing)
;;(start-simulating-time!)
(def cluster (mk-local-storm-cluster))
(use 'backtype.storm.bootstrap) (bootstrap)
(import '[backtype.storm.testing TestWordCounter TestWordSpout TestGlobalCount TestAggregatesCounter])
(def spout (feeder-spout ["word"]))
(def topology (thrift/mk-topology
{1 (thrift/mk-spout-spec spout :parallelism-hint 3)}
{2 (thrift/mk-bolt-spec {1 ["word"]} (TestWordCounter.) :parallelism-hint 4)
3 (thrift/mk-bolt-spec {1 :global} (TestGlobalCount.))
4 (thrift/mk-bolt-spec {2 :global} (TestAggregatesCounter.))
}))
(submit-local-topology (:nimbus cluster) "test" {TOPOLOGY-WORKERS 4 TOPOLOGY-DEBUG true} topology)
* clean up project
- remove log4j dir and instead generate it in the deploy (it's only used in bin/storm -> create a console one and put into bin/)
- include system component / stream information in the topologycontext and clean up system specific code all over the place
* Very rare errors
weird nullptr exceptions:
(tasks i) on send-fn
no virtual port socket for outbound task (in worker)