The Social Network

The Social Network

Sean Parker: We lived in farms, then we lived in cities, and now we're gonna live on the internet!

Erlang at Facebook

These are some excerpts from a presentation about the Facebook chat implementation.

System challenges
- How does synchronous messaging work on the Web?

- "Presence" is hard to scale
-- Need a system to queue and deliver messages
-- Millions of connections, mostly idle
-- Need logging, at least between page loads

(...)

Architectural details of Channel servers
- Distributed design
- User id space is partitioned (division of labor)
-- Each partition is serviced by a cluster (availability)
- Presence aggregation
-- Channel servers are authoritative
-- Periodically shipped to presence servers
- Open source: Erlang, Mochiweb, Thrift, Scribe, fb303, et al

(...)

Key Erlang Features we love:
- Concurrency (based on user-mode threads)

- Distribution Connected network of nodes
-- Remote processes look like local processes
-- Any node in a channel server cluster can route requests
-- Naive load balancing

- Fault Isolation

- Error logging
-- Stacktraces point the way to bugs (functional languages win big here)

- Hot code swapping

- Monitoring and Error Recovery
-- Supervision hierarchies
-- Organize (and control) processes
-- Organize thoughts
-- Systematize restarts and error recovery
-- simple_one_for_one for dynamic child processes

- net_kernel (Distributed Erlang)
-- sends nodedown, nodeup messages
-- any process can subscribe
-- heart: monitors and restarts the OS process

- Remote Shell
-- Ad-hoc inspection of a running node
-- Command-and-control from a console
-- Combines with hot code loading

- Erlang top (etop)
-- Shows Erlang processes, sorted by reductions, memory and message queue
-- OS-like functionality ... for free

- Hibernation

- Symmetric MultiProcessing (SMP)
-- Take advantage of multi-core servers
-- erl -smp runs multiple scheduler threads inside the node
-- SMP is emphasized in recent Erlang development

- hipe_bifs
-- hipe_bifs:bytearray_update() allows for destructive array assignment
-- Cheating single assignment because in Erlang is destructive assignment is hard because it should be

Reference: Erlang at Facebook by Eugene Letuchy on Apr 30, 2009