Mesh Splitting
MOOSE provides the ability to pre-split a mesh into several chunks for use with distributed mesh. This can be useful if the whole mesh is too big to fit into memory. A split-mesh workflow involves generating the mesh split configuration(s) and then telling moose to use them when you run simulations. This is described in the sections below.
Generating Split Configurations
You can run MOOSE with the --split-mesh <split1>[,<split2>]... flag which takes a comma-separated list of describing each split with an integer number of chunks to split the mesh into.
$ moose-app-opt -i your_input.i --split-mesh 13,42,1000
Splitting 13 ways...
- writing 13 files per process...
Splitting 42 ways...
- writing 42 files per process...
...
This will create three split configurations for your mesh that include 13, 42, and 1000 chunks each. If your input file uses the mesh file foo.e, this process will generate a directory named foo.cpr. In general, you should neither rename this directory nor modify its contents. All split configurations for the mesh will be stored in this directory (even across independent splitting operations).
For non-file-based mesh cases (e.g. GeneratedMesh), you will need to tell MOOSE what name to use for the split mesh files it will generate using the --split-file <file_name> flag; in general, you should use the .cpr file extension here. For example:
$ moose-app-opt -i your_input.i --split-mesh 42 --split-file foo.cpr
You can also run the splitting operation itself in parallel on several processors to speed up the operation:
$ mpiexec -n 9 -i moose-app-opt your_input.i --split-mesh 42
Splitting 42 ways...
- writing 5 files per process...
This will generate the same split configuration as the 42 chunk split generated by the first command but just generates it in parallel.
It is noted that if there are mesh meta data generated by mesh generators, these meta data will be written to a binary file under the generated directory that can be loaded when using split meshes.
Using Split Meshes
To use a mesh split configuration use the --use-split flag (which takes no arguments):
$ mpiexec -n 42 moose-app-opt -i your_input.i --use-split
You may also use the Mesh/use_split input parameter.
This will cause MOOSE to look for a mesh split configuration with 42 chunks. If one exists, MOOSE will use it running in a distributed mesh mode, otherwise an error will occur. Note that you do not need to modify your input file - MOOSE automatically switches to using the pre-split distributed mesh regardless of the mesh type specified in the input file. If your input file does not use a file-based mesh, you will need to specify a split mesh file name using the --split-file <file_name> flag just as you did when splitting the mesh:
$ mpiexec -n 42 moose-app-opt -i your_input.i --use-split --split-file foo.cpr
The mesh splitter commands do not work with DistributedMesh. You must only split with a ReplicatedMesh.
Using --use-split skips the [Mesh] block of the input file. Split mesh may be loaded using a FileMeshGenerator if further mesh generation is to be performed on the split mesh.