GProc

GProc - Erlang Global Process Registry

2009.09.16

First blog post from my new position at Basho Technologies, home of Riak, a fast, flexible, scalable, operationally-easy data store written in Erlang. In this post, I examine gproc, an improved global process registry written in 2007 by Ulf Wiger, for possible use by Riak and Nitrogen.

The Erlang Process Registry

Very often in Erlang, you want to create a process and give it a name. This allows you to send messages to that process from other processes without passing the Pid around. Erlang supports named processes using erlang:register(Name, Pid), but there are limitations.

The Name must be an atom.
A process can only have one, and a name can be associated with only one process.
The local and global registries are separate. (A process is registered globally via global:register_name(Name, Pid).)
There can be a lag between when a process is registered, and when it can receive messages under its registered name. ie: calling "register(myprocess, self())", and then immediately calling "myprocess ! message", can cause an error.

These limitations constrain the way Erlang developers architect and debug applications. With a more robust process registry capable of storing additional metadata about each registered process, a developer can better decouple the responsibilities of an application, or gather information about running processes for debugging or reporting purposes.

Some concrete examples:

Riak spins off a new process for each get, put, and delete operation. There is currently no way to tell how many data operations are running at any one time, or to look inside a process once it is running to see what bucket it will modify. Ideally, this could be attached as metadata to the process, and a separate monitoring application could fetch this data and display it on the Riak Web dashboard.

Nitrogen spins off a process for a user's session. Ideally, on subsequent requests Nitrogen could look up the session process in the registry based on the user's cookie. For now, a separate lookup structure is used.

Ulf Wiger, back in his days at Ericcson (he is currently the CTO of Erlang Training and Consulting, noticed that he and other engineers were repeatedly (re)creating their own process registry structures in the form of lists and lookup tables as a way of working around the limitations described above.

To solve the problem in a more general fashion, Ulf created gproc. (Well, he first created proc, and then--with the insight he gained--created gproc.)

gproc lets you:

associate one or more names with a process, where a name can be any Erlang term
attach one or more arbitrary properties to a process, where the key or value can be any Erlang term
associate a counter with a process (with an optional rollup to aggregate counters across multiple processes), where the process name can be any Erlang term
retrieve any of these keys and values using ETS style wildcard matching

gproc can be confusing at first because naming processes, applying properties, and creating counters are all accomplished using gproc:reg(GProcKey, Value). This was done in the name of speed, consistency, code-reuse -- all of these operations store data in the same ETS table, and so gproc makes functions double up.

Furthermore, to ensure thread safety, a process can only set it's own values. To put it another way, it's as if erlang:register(Name, Pid) became erlang:register(Name) and can only act on the current process.

Finally, it helps to think of "registering" a process as instead "the process sets a globally unique key". Similarly, "setting a property" means "the process sets a non-unique key".

You'll see what I mean in the examples below.

Start a shell with GProc

To get a gproc playground, run the following:

    svn checkout http://svn.ulf.wiger.net/gproc/branches/experimental-0906/gproc
    cd gproc
    erlc -I include -o ebin src/*.erl
    erl -pa ebin
    &gt; application:start(sasl).
    &gt; application:start(gproc).

Storing a value

So let's play. To store a value in gproc, call "gproc:reg({n, l, Key}, Value)"

Again, both Key and Value can be any Erlang term. With a type of n (for name) you can only associate one value with a key. Try it again, and gproc will throw an exception.

Removing a Value

To remove something from gproc, call "gproc:unreg({n, l, Key})". Alternatively, kill the process. gproc monitors all processes that have stored values. When a process dies, gproc automatically removes any values associated with that process. This can lead to some confusion when testing gproc in the Erlang shell. If you store some values in gproc, and then run a gproc operation that throws an exception, Erlang restarts your shell process, so you lose any gproc values associated with that process.

Though this can be confusing in the shell, this is exactly the behaviour we want from a process registry.

Retrieving a Value

To get your stored value, call "gproc:get_value({n, l, Key})". Note that this will only retrieve values stored by the current process. You can also use "gproc:get_value({n, l, Key}, Pid)" to get values from another pid.

Who put the cookie in the cookie jar?

In addition to your value, gproc stores the Pid responsible for the value. You can retrieve this by calling "gproc:lookup_pid({n, l, Key})" or "gproc:lookup_pids({p, l, Key})". (You use the former when working with registered properties and aggregated counters, and the latter when working with properties and local counters.)

Taking a step back

So you are probably wondering about the significance of the 3-tuple that we are passing to these functions. This tuple, of the form {Type, Scope, Key} tells gproc a few things about your value:

Type is either 'n', 'p', 'c', or 'a' (n = name, p = property, c = counter, a = aggregate counter)
Scope is either 'l' or 'g' ('l' = local, 'g' = global)
Key is any Erlang term.

So far, we have been using a type of 'n', a globally unique key, which gives us a good mechanism for uniquely naming a process. This is similar to erlang:register/2.

Specifying 'p', for property, lets us set a key/value pair for a process, but other processes can also set a name and value of their own.

Specifying 'c', for counter, lets you set and increment a local counter for a process.

And specifying 'a' for aggregated counter, lets you access the sum of all local counters by the same name.

Note: we're always using 'l' for local scope. See the 'G is for Global' note below on why I haven't written anything about global scope.

Registering a Process

To register a process in gproc, call "gproc:reg({n, l, Key}, ignored)". The 'ignored' atom is irrelevant here, but could be used to store some extra metadata about the process.

Now you can use:

gproc:lookup_pid({n, l, Key}) or gproc:where({n, l, Key}) to return the Pid of the process.
gproc:send({n, l, Key}, Msg) to send a message to the process.

These functions are also available for registered processes, but less useful:

gproc:get_value({n, l, Key}) gets the value from the process that set it.
gproc:select/1 to gets the value from any process (more below).

Setting a Property

To set a property, call "gproc:reg({p, l, Key}, Value)".

Now, you can use:

gproc:get_value({p, l, Key}) to get the value from the process that set it.
gproc:select/1 to get the value from any process (more below).
gproc:lookup_pids({p, l, Key}) to return a list of processes matching the key.
gproc:send{p, l, Key}) to send a message to all processes matching the key.

Creating a Local Counter

To create a local counter, call "gproc:reg({c, l, Key}, Integer)".

Now, you can use:

gproc:update_counter({c, l, Key}, Integer) to increment or decrement the counter.
gproc:select/1 to get the counter value from any process (more below).
gproc:lookup_pids({c, l, Key}) to return a list of processes matching the key.
gproc:send{c, l, Key}) to send a message to all processes matching the key.

Creating an Aggregated Counter

To create an aggregated counter, call "gproc:reg({a, l, Key}, Ignored)"

Now you can use:

gproc:get_value({a, l, Key}) to get the counter value from the process that set it.
gproc:select/1 to get the counter value from any process (more below).
gproc:lookup_pid({n, l, Key}) or gproc:where({n, l, Key}) to return the Pid of the process that set the counter.
gproc:send({n, l, Key}, Msg) to send a message to the process that set the counter.

It's Really a Shared Dictionary

As you can see, gproc is really a shared dictionary with some limitations on who can store what when. Again, this leads to some confusion at first, but means that gproc can be applied to a wider variety of problems.

gproc:select/1

So far, we've glossed over gproc:select/1, and for good reason: it's complicated. But, it's also where gproc derives most of its power, so it pays to understand it.

With gproc:select/1, you can query the gproc process registry using roughly the same semantics as an ETS MatchSpec. Here is a quick (and far from complete) tutorial:

The basic form is gproc:select(Scope, MatchSpec).

Scope can either be 'all', 'names', 'props', 'counters', or 'aggr_counters'. MatchSpec is of the form [{MatchHead, Guard, Result}]. (Yes, it must be wrapped within a list.)

The MatchHead is a 3-tuple of the form {GProcKey, Pid, Value}. GProcKey is the 3-tuple key we've seen earlier, ie: "{n, l, Key}". By leaving an empty Guard ([]) for now, and using a catch-all Result (['$$']), we can already do some interesting things.

This will pull out every record in the process registry:

    MatchHead = '_',
    Guard = [],
    Result = ['$$'],
    gproc:select([{MatchHead, Guard, Result}]).

This will pull out every record where value == value1:

    MatchHead = {'_', '_', value1},
    Guard = [],
    Result = ['$$'],
    gproc:select([{MatchHead, Guard, Result}]).

You can specify a more complicated keys. This example will pull out any entry whose key would match "{myrecord, _}"

    Key = {myrecord, '_'},
    GProcKey = {'_', '_', Key}
    MatchHead = {GProcKey, '_', '_'},
    Guard = [],
    Result = ['$$'],
    gproc:select([{MatchHead, Guard, Result}]).

MatchHead can also use '$1' variables, which lets you start using guards to shape your select. For example, this will pull out any gproc entry whose key is a list:

    GProcKey = {'_', '_', '$1'},
    MatchHead = {GProcKey, '_', '_'},
    Guard = [{is_list, '$1'}],
    Result = ['$$'],
    gproc:select([{MatchHead, Guard, Result}]).

See this gist for more examples. http://gist.github.com/188032.

Normally, you would be able to use the Result property of the MatchSpec to select out a subset of the data you want to capture. In gproc this is slightly broken. You can choose which result you want to pull out by setting result to a '$1' variable (for example, ['$1']), but currently you can only pull out one result from each match. In other words, ['$1', '$2'] does not work. (It only returns '$2'.)

GProc and QLC

It appears that GProc began to have support for QLC, which is pretty friggin' sweet, but unfortunately I could not seem to make it work.

G is for Global

GProc automatically detects whether gen_leader is available on the system. If it is, then gproc will run in "global mode" meaning that all of the functions above work across your entire Erlang cluster rather than on a single node, which is also pretty friggin' sweet. Unfortunately, gproc currently uses an obsolete version of gen_leader, so running gproc in global mode is not recommended.

Evaluation

GProc is extremely powerful, but the interface could use simplification. Convenience methods around gproc:reg/2 specifically shaped for registering processes would go far to help smooth the learning curve for newcomers.

In addition, gproc:select/1 has a few small bugs. The MatchSpec must be passed as a list, even though it will only work with one MatchSpec, and the Result portion of the MatchSpec can only be used with one '$N' variable.

Update from Ulf Wiger:

Gproc has now been moved to Github: http://github.com/uwiger/gproc.

I have added some functions, such as gproc:await(UniqueName, Timeout), which will suspend until the name key UniqueName becomes available, and then return a {Pid, Value} tuple representing the registration info. One way to use this is as a simple resource broker, or to manage dependencies during system start.

I have done some initial testing of the global parts of gproc. They seem to work. I used the hanssv+sergeversion at http://github.com/uwiger/genleaderrevival in which I have made small changes. The other genleader versions should work as well, but I haven't tested them. See gprocdist:elected/[2,3] to understand the differences (the hanssv+sergeversion calls elected/3, while the others call elected/2).