3.4 MAIN

The programs are essentially the same as the serial version until the initial data set is read from the input file. Throughout this program, when I/O has to be done, it is only done by the root node (rank=0). Dynamic allocation of several arrays based on the problem size (nx, ny, nz) as well as those based on nphase also occur here. There are some small difference in details between the finite element and finite difference programs, which will be obvious when looking more closely at the programs.

• Initialize variables.
• Initialize MPI.
• Root calculates z extents, d1 and d2.
• Root reads in original data set and passes appropriate sections to other processing nodes.
• Dynamically allocate large arrays per processor.
• Call FEMAT_MPI - this computes the local stiffness matrices and any other auxiliary arrays needed. The local stiffness matrices are used in ENERGY_MPI and DEMBX_MPI to construct the global stiffness matrix as the various arrays are updated in the conjugate gradient process.
• Calculate initial u array per processor.
• Call ENERGY_MPI - this computes the initial energy of the system, based on the initial conditions, and also computes the initial value of gb, the gradient of the energy array. Both of these are needed as initial inputs to DEMBX_MPI.
• Enter loop which calls DEMBX_MPI and ENERGY_MPI. DEMBX_MPI is the conjugate gradient subroutine that actually performs the conjugate gradient update to the variables of the problem. It is performed ldemb number of times, and is then exited and ENERGY_MPI is then called again. This is done in order to see how the relaxation is going, and whether the energy has relaxed enough. It is usually not possible, a priori, to set the value of gtest so that good results are obtained in a reasonable amount of time. If gtest is set too small, many conjugate gradient cycles could be wasted to try to push the accuracy of the answer beyond what is necessary. If gtest is too large, then the final answer obtained might not be accurate enough.
• After convergence, calculate final results, i.e. stress, strain, or current per pixel and then produce a global summation.