HipG

High-level Distributed Processing of Large-Scale Graphs

HipG is a library for high-level parallel processing of large-scale graphs. HipG is implemented in Java and is designed for distributed-memory machine. Besides basic distributed graph algorithms it handles divide-and-conquer graph algorithms and algorithms that execute on graphs created on-the-fly. It is designed for clusters of machines, but can also be tested on desktops - all you need is a recent Java runtime environment. HipG is work in progress! (as of Apr'11)

download | building and testing | local runs | cluster runs | documents | credits

Download

Download HipG here:

The bundle contains:

To run HipG program you only need Java 6. Dependencies come bundled, many input graphs can be created in memory, and you can run either on your desktop or on a cluster.

Building and Testing

Building is not necessary, unless you want to modify the code, in which case ant is needed. Unpack sources and run 
   ant
Debugging can be done with Log4j We tested under Linux, using JVM v. 6 by Sun/Oracle. In particular, the library is not yet tested under Windows.

Licence

HipG is licensed under GPL. For commercial use, please contact the authors.

Running on a Local Machine

Before executing any HipG program, start a server: 

    ./bin/server.sh
to which all workers will connect and obtain a rank. The server does not at all take part in computations. Each set of workers is called a pool and the server can handle many pools at the same time.

Examples used in papers include

    hipg.app.(Visitor|BFS|scc.FB|scc.owctypa3|onthefly.SpinJadi)
In general, to run a HipG application *locally* you typically type: 
./bin/run-local.sh <#threads> <program> <graph>
To run, for example, the Visitor application *locally* using 4 threads on a binary tree graph of height 20, execute 
    ./bin/run-local.sh 4 hipg.app.Visitor BinTree 20
The tree will be first created and randomly partitioned between the memory of workers.

Running on a Cluster

Before executing any HipG program, start a server on one of the workers, or on a head node, or on any other machine. Supply the run script with a -host option to point to the machine that runs the server. The server has little impact on performance of the machine that runs it.

Use the script ./bin/run.sh to run HipG jobs on a cluster. The script invokes prun, which submits a job to, in our case, an SGE scheduler. Substitute your own prun or simply edit the run script to invoke something else. An example of a cluster run is: 

    ./bin/run.sh -host <head.node.edu> 16 hipg.app.Visitor BinTree 26

Documents

The HipG paper was published at ICDCN in Jan 2011. The technical report is an extended version of this paper:

Have fun with HipG!

Contributors to the project: Questions? Problems? Contact Ela at .

Amsterdam, 20 Apr 2011