# Parallel runs with MPI
To compile with MPI parallelism enabled, you need to use something like:
~~~bash
CC99='mpicc -std=c99' qcc -Wall -O2 -D_MPI=1 example.c -o example -lm
~~~
where *mpicc* calls the MPI compiler on your system. The resulting
executable can then be run in parallel using something like
~~~bash
mpirun -np 8 ./example
~~~
The details may vary according to how the MPI compiler is setup on
your system.
## Using Makefiles
The "manual" way above is automated if you use the [standard
Makefiles](/Tutorial#using-makefiles) provided by Basilisk. You can
then compile and run the example above on eight processes using:
~~~bash
CC='mpicc -D_MPI=8' make example.tst
~~~
This assumes that *mpicc* and *mpirun* are available on your system.
## Running on supercomputers
A simple way to run Basilisk code on a supercomputer is to first
generate a portable (ISO C99) source code on a machine where *qcc* is
installed i.e.
~~~bash
%localmachine: qcc -source -D_MPI=1 example.c
~~~
Then copy the portable source code *_example.c* (don't forget the
underscore!) on the supercomputer and compile it:
~~~bash
%localmachine: scp _example.c login@supercomputer.org:
%localmachine: ssh login@supercomputer.org
%supercomputer.org: mpicc -Wall -std=c99 -O2 -D_MPI=1 _example.c -o example -lm
~~~
where the *-std=c99* option sets the version of the language to C99. Note that
this option may change depending on the compiler (the options shown above are
valid for *gcc* or *icc*, the Intel compiler).
You will then need to use the job submission system of the
supercomputer to set the number of processes and run the
executable. See also the following examples:
* [Parallel scalability](test/mpi-laplacian.c#how-to-run-on-occigen)
* [Atomisation of a pulsed liquid jet](examples/atomisation.c#running-in-parallel)
* [Forced isotropic turbulence in a triply-periodic box](examples/isotropic.c#running-with-mpi-on-occigen)
## Non-cubic domains
For the moment, the only way to combine non-cubic domains and MPI
parallelism implies using multigrids rather than tree grids (because
[masking](/Basilisk%20C#complex-domains) does not work together with
MPI yet). This also means that MPI-parallel, non-cubic and adaptive
simulations are not possible yet.
On multigrids, MPI subdomains are setup using Cartesian topologies
i.e. the processes are arranged on a line (in 1D), a rectangle (in 2D)
or a [cuboid](https://en.wikipedia.org/wiki/Cuboid) (in 3D). The total
number of processes $n_p$ thus verifies the relation $n_p = n_x n_y
n_z$ where $n_x$, $n_y$, $n_z$ are the number of processes along each
axis. Controlling the number of processes along each axis allows to
change the aspect ratio of the (global) domain.
If $n_x$, $n_y$ and $n_z$ are not specified by the user, Basilisk sets
them automatically based on the value of $n_p$ (as set by the `mpirun
-np` command). To do so, it calls the
[`MPI_Dims_create()`](https://www.open-mpi.org/doc/v1.8/man3/MPI_Dims_create.3.php)
MPI function. Some particular cases are:
* $n_p$ is a [square
number](https://en.wikipedia.org/wiki/Square_number) (in 2D) or
[cubic number](https://en.wikipedia.org/wiki/Cubic_number) (in 3D):
In this case $n_x = n_y = n_z = n_p^{1/3}$ i.e. the domain is a
cube.
* $n_p$ is a prime number: $n_x = n_p$, $n_y = n_z = 1$ i.e. the
domain is a long channel.
* $n_p$ is the product of two prime numbers: a 3D decomposition is not
possible, an MPI error is returned.
To explicitly control the dimensions along each axis, one can call the
`dimensions()` function. Any dimension which is not set is computed
automatically.
All this is also compatible with global [periodic
boundaries](/Basilisk%20C#periodic-boundaries) (but not periodic
boundaries for individual fields).
### Physical dimensions and spatial resolution
As usual, the physical dimension of the domain is set by calling the
`size()` function. What is set is the size along the x-axis of the
entire domain (not of the individual subdomains). This allow one to
vary the number of processes while keeping the physical size constant
(i.e. do the same simulation on a different number of processes).
The spatial resolution (N) is handled in a similar way, with the added
constraint that it must be a multiple of $n_x\times 2^n$ with $n$ an
integer.
### Examples
A $[0:1]\times[0:1/3]\times[0:1/3]$ channel, periodic along the x-axis
(with slip walls), on 3 MPI processes, with 96 points along the
x-axis:
~~~literatec
#include "grid/multigrid3D.h"
...
int main() {
periodic (right);
init_grid (128);
...
}
~~~
The same simulation but on 24 processes:
~~~literatec
#include "grid/multigrid3D.h"
...
int main() {
dimensions (nx = 6);
periodic (right);
init_grid (128);
...
}
~~~
A cube centered on the origin, of size 1.5, all-periodic, with 512
points along each axis, on 8 = 2^3^ or 64 = 4^3^ or 512 = 8^3^ or 4096
= 16^3^ etc... processes:
~~~literatec
#include "grid/multigrid3D.h"
...
int main() {
size (1.5);
origin (-0.75, -0.75, -0.75);
foreach_dimension()
periodic (right);
init_grid (512);
...
}
~~~