ANSYS FENSAP-ICE Installation and Licensing

Guide

ANSYS, Inc. Release 2020 R1

Southpointe January 2020
2600 ANSYS Drive
Canonsburg, PA 15317 ANSYS, Inc. and
[email protected] ANSYS Europe,
Ltd. are UL
http://www.ansys.com registered ISO
(T) 724-746-3304 9001: 2015
(F) 724-514-9494 companies.
Copyright and Trademark Information

© 2020 ANSYS, Inc. Unauthorized use, distribution or duplication is prohibited.

ANSYS, ANSYS Workbench, AUTODYN, CFX, FLUENT and any and all ANSYS, Inc. brand, product, service and feature
names, logos and slogans are registered trademarks or trademarks of ANSYS, Inc. or its subsidiaries located in the
United States or other countries. ICEM CFD is a trademark used by ANSYS, Inc. under license. CFX is a trademark
of Sony Corporation in Japan. All other brand, product, service and feature names or trademarks are the property
of their respective owners. FLEXlm and FLEXnet are trademarks of Flexera Software LLC.

Disclaimer Notice

THIS ANSYS SOFTWARE PRODUCT AND PROGRAM DOCUMENTATION INCLUDE TRADE SECRETS AND ARE CONFID-
ENTIAL AND PROPRIETARY PRODUCTS OF ANSYS, INC., ITS SUBSIDIARIES, OR LICENSORS. The software products
and documentation are furnished by ANSYS, Inc., its subsidiaries, or affiliates under a software license agreement
that contains provisions concerning non-disclosure, copying, length and nature of use, compliance with exporting
laws, warranties, disclaimers, limitations of liability, and remedies, and other provisions. The software products
and documentation may be used, disclosed, transferred, or copied only in accordance with the terms and conditions
of that software license agreement.

ANSYS, Inc. and ANSYS Europe, Ltd. are UL registered ISO 9001: 2015 companies.

U.S. Government Rights

For U.S. Government users, except as specifically granted by the ANSYS, Inc. software license agreement, the use,
duplication, or disclosure by the United States Government is subject to restrictions stated in the ANSYS, Inc.
software license agreement and FAR 12.212 (for non-DOD licenses).

Third-Party Software

See the legal information in the product help files for the complete Legal Notice for ANSYS proprietary software
and third-party software. If you are unable to access the Legal Notice, contact ANSYS, Inc.

Published in the U.S.A.

Table of Contents
1. NTI Licensing ........................................................................................................................................... 1
1.1. License Server Setup ......................................................................................................................... 1
1.2. The License File ................................................................................................................................. 1
1.3. Client Setup ...................................................................................................................................... 2
1.4. Server Setup ..................................................................................................................................... 2
1.4.1. Windows .................................................................................................................................. 2
1.4.2. Linux ....................................................................................................................................... 4
1.5. License Not Available ........................................................................................................................ 5
1.6. Licensing - Server Hostname ............................................................................................................. 5
1.7. Licensing - Server Host ID ................................................................................................................. 6
1.8. Licensing - Server Version ................................................................................................................. 6
1.9. Licensing – Reset Settings ................................................................................................................. 6
2. MPI .......................................................................................................................................................... 9
2.1. Network Types .................................................................................................................................. 9
2.2. Intel MPI ......................................................................................................................................... 10
2.2.1. Platform Notes ....................................................................................................................... 10
2.3. MPI Setup in FENSAP-ICE ................................................................................................................ 10
2.3.1. FENSAP-ICE Advanced MPI Configuration ............................................................................... 10
2.3.1.1. Override mpirun ............................................................................................................ 11
2.3.1.2. Additional mpirun Parameters ....................................................................................... 11
2.3.2. Machinefile ............................................................................................................................ 12
2.3.3. Per-Solver machinefile ............................................................................................................ 12
2.3.4. Saving the Configuration to the Project .................................................................................. 13
3. Queuing Systems .................................................................................................................................. 15
3.1. PBS ................................................................................................................................................. 15
3.1.1. $PBS_NODEFILE ..................................................................................................................... 15
3.1.2. PBS Job Settings ..................................................................................................................... 15
3.1.3. The qsub.cfg file ..................................................................................................................... 17
3.1.4. Special Files ........................................................................................................................... 18
3.2. SGE Queue ..................................................................................................................................... 18
3.2.1. SGE Job Settings .................................................................................................................... 18
3.2.2. Special Files ........................................................................................................................... 19
3.3. "NULL" Queue ................................................................................................................................. 20
3.4. "CUSTOM" Queue ........................................................................................................................... 20
3.4.1. "CUSTOM" Queue for Windows ............................................................................................... 22
4. Advanced MPI Topics ............................................................................................................................. 23
4.1. Software Licenses and Parallel Jobs ................................................................................................. 23
4.2. Sweep, Langmuir, FENSAP-ICE-TURBO and Machinefiles ................................................................... 23
5. Configuration Options .......................................................................................................................... 27
5.1. config/gui.txt (Linux only) ............................................................................................................... 27
5.2. config/mpi.txt ................................................................................................................................. 27
5.3. config/licensing.txt ......................................................................................................................... 28
6. Directory Structure ............................................................................................................................... 29
6.1. Installation Directory ...................................................................................................................... 29
6.2. User Account – Linux ...................................................................................................................... 30
6.3. User Account - Windows ................................................................................................................. 30

Release 2020 R1 - © ANSYS, Inc. All rights reserved. - Contains proprietary and confidential information
of ANSYS, Inc. and its subsidiaries and affiliates. iii
 Release 2020 R1 - © ANSYS, Inc. All rights reserved. - Contains proprietary and confidential information
iv of ANSYS, Inc. and its subsidiaries and affiliates.
List of Tables
1.1. License Server ........................................................................................................................................ 2
2.1. Topologies .............................................................................................................................................. 9

Release 2020 R1 - © ANSYS, Inc. All rights reserved. - Contains proprietary and confidential information
of ANSYS, Inc. and its subsidiaries and affiliates. v
 Release 2020 R1 - © ANSYS, Inc. All rights reserved. - Contains proprietary and confidential information
vi of ANSYS, Inc. and its subsidiaries and affiliates.
Chapter 1: NTI Licensing
The following sections of this chapter are:
1.1. License Server Setup
1.2.The License File
1.3. Client Setup
1.4. Server Setup
1.5. License Not Available
1.6. Licensing - Server Hostname
1.7. Licensing - Server Host ID
1.8. Licensing - Server Version
1.9. Licensing – Reset Settings

1.1. License Server Setup

Note:

These setup instructions are specific to legacy NTI license files. These setup instructions are
specific to legacy NTI license files. ANSYS license files should be installed and managed using
the ANSYS License Manager.

On the machine acting as the license server, provide the hostname, and the Ethernet MACAddress of
the machine.

• Windows:

– Launch LMTOOLS (provided in the license/ directory of FENSAP-ICE). And in the System settings, select
Save HOSTID to file.

• Linux:

– From the license/ directory of the FENSAP-ICE installation, execute the command: > ./lmutil
lmhostid

1.2. The License File

The license file provided for FENSAP-ICE contains important lines in the header:

SERVER hostname 000123456789 1060

USE_SERVER

Release 2020 R1 - © ANSYS, Inc. All rights reserved. - Contains proprietary and confidential information
of ANSYS, Inc. and its subsidiaries and affiliates. 1
NTI Licensing

VENDOR NTI port=60824

The italicized sections can be edited to select the ports used by the license server.

Table 1.1: License Server

Hostname Edit this field if the hostname of the machine is

changed.
1060 Optional, the default port is 27000. In case of
a conflict with another FLEXlm service on the
machine, use a different port. This port will be
used in the port@hostname when setting
up the FENSAP-ICE licensing.
port=60824 Optional, used to specify a fixed address on
the second port of the licensing system. Use if
specific ports need to be enabled on the
network firewall.

1.3. Client Setup

ANSYS licensing is used by default. To switch to NTI licensing, the NTI_LICENSE_FILE variable must
be set up. Either as:

• Environment variable: Set the NTI_LICENSE_FILE=port@server variable.

• <install_dir>/config/licensing.txt (See config/licensing.txt (p. 28)).

1.4. Server Setup

In this section you will set up, run and save a FLEXlm server.

1.4.1. Windows
1. Open lmtools.exe from the …/license/ folder of FENSAP-ICE

2. Set up the path to lmgrd.exe, license file and output log file in the Config Services tab

Release 2020 R1 - © ANSYS, Inc. All rights reserved. - Contains proprietary and confidential information
2 of ANSYS, Inc. and its subsidiaries and affiliates.
 Server Setup

3. Save Service

To enable the license server automatic restart if the server is rebooted, use the Use Services check
box.

4. Start the server in the Start/Stop/Reread tab.

5. To check for any license activation error, check the log. View Log in the Config Services tab.

Release 2020 R1 - © ANSYS, Inc. All rights reserved. - Contains proprietary and confidential information
of ANSYS, Inc. and its subsidiaries and affiliates. 3
NTI Licensing

Note:

If other FLEXlm servers are already running on the same machine, you might not be
able to start the server on the default port. In that case, specify an additional value
(the port number) on the server line of the license file.

1060 is used in the following example:

SERVER put_machinename_here 000000000 1060

Later, when identifying the server, the network address will be 1060@machinename

1.4.2. Linux
The license server is launched by executing the following command from the …/license/ directory
of FENSAP-ICE:

> lmgrd –c nti.lic –l nti.log

(nti.lic is the filename of the license file, nti.log will contain the license server log and error
messages).

On a dedicated license server, this daemon should be scheduled to be automatically executed at

boot time.

The license server can be stopped with the command:

> lmutil lmdown –c nti.lic

Additional diagnostic and utility functions are available in the lmutil tool.

Release 2020 R1 - © ANSYS, Inc. All rights reserved. - Contains proprietary and confidential information
4 of ANSYS, Inc. and its subsidiaries and affiliates.
 Licensing - Server Hostname

1.5. License Not Available

Issue: The license server is not started, not reachable or misconfigured.

Solution:

• Server not started (Windows):

Open LMTOOLS and start the license server. Make sure that the Use Services and Start Server at
Power Up in the Config Services section are enabled; these setting will ensure that the server will
restart when the machine is rebooted. Check the log file (View Log button in the Config Services)
to see if the server has started cleanly with no errors.

• Server Not Reachable:

If the server is on a remote machine, ensure that it can be reached and there are no firewall policies
that prevent access.

• Server Misconfigured:

Check the license log file and the next sections of this document for an explanation of common errors
that can occur.

1.6. Licensing - Server Hostname

FLEXlm log:

14:09:17 (lmgrd) "NAME1": Not a valid server hostname,

Release 2020 R1 - © ANSYS, Inc. All rights reserved. - Contains proprietary and confidential information
of ANSYS, Inc. and its subsidiaries and affiliates. 5
NTI Licensing

exiting.

14:09:17 (lmgrd) Valid license server system hosts are:

"NAME2"

Issues: The computer hostname recognized by the license system is NAME2, but the license file states
NAME1.

Solution: Edit the license file, correcting the SERVER line hostname

1.7. Licensing - Server Host ID

FLEXlm log:

14:24:22 (NTI) Wrong hostid on SERVER line for license

file:

14:24:22 (NTI) E:\NTI\nti_wronghostid.lic

14:24:22 (NTI) SERVER line says HOST1, hostid is HOST2

14:24:22 (NTI) Invalid hostid on SERVER line

Issue: The license file is not suited for this machine, or the network cards were changed or disabled.

Solution: Contact support.

1.8. Licensing - Server Version

FENSAP-ICE startup message:

Communication error - wrong FLEXlm server version - cannot checkout

licenses

: Bad encryption handshake with vendor daemon.

Feature: FENSAPICE

Issue: The version of the software is mismatched with the server/license version.

Solution: Ensure you are using the software and the license combo provided with this release of FENSAP-
ICE.

1.9. Licensing – Reset Settings

Typically not required. In order to reset any previous settings for the license, the following can be done:

• Windows: Reset the FENSAP-ICE license file settings for the current user.

– Open REGEDIT (type regedit in the Run section of the Start menu).

Release 2020 R1 - © ANSYS, Inc. All rights reserved. - Contains proprietary and confidential information
6 of ANSYS, Inc. and its subsidiaries and affiliates.
 Licensing – Reset Settings

– Search for the NTI_LICENSE_FILE entry.

– Erase it.

• Linux: Remove the NTI_LICENSE_FILE line from ~/.flexlmrc

Release 2020 R1 - © ANSYS, Inc. All rights reserved. - Contains proprietary and confidential information
of ANSYS, Inc. and its subsidiaries and affiliates. 7
 Release 2020 R1 - © ANSYS, Inc. All rights reserved. - Contains proprietary and confidential information
8 of ANSYS, Inc. and its subsidiaries and affiliates.
Chapter 2: MPI
In order to execute in parallel on multi-processor machines and speed-up the computation, FENSAP-
ICE runs within the MPI communication framework. The software packages that can be run in parallel
are:

• FENSAP

• DROP3D

• ICE3D

• C3D

• OptiGrid

MPI is a commonly available communication system that enables multiple copies of the same executable
to run in parallel, either on the same machine or over a network of machines. All copies work on a
separate part of the problem and communicate with each other to exchange data and solve the overall
problem.

2.1. Network Types

There are three main types of MPI topologies:

Table 2.1: Topologies

A single computer. Single CPU or multi-cores.

Communication system: Local (Shared

memory).
Multiple computers on a network.

Communication system: Ethernet.

Release 2020 R1 - © ANSYS, Inc. All rights reserved. - Contains proprietary and confidential information
of ANSYS, Inc. and its subsidiaries and affiliates. 9
MPI

Computer cluster, with queuing system.

Communication system: Ethernet / Infiniband

/ Myrinet or other high-speed, dedicated
network.

Even if the various modules of FENSAP-ICE are used on a single computer, the MPI environment must
still be properly configured. FENSAP-ICE bundles Intel MPI in the installation package and therefore no
configuration is required. This document addresses advanced configurations and alternate MPI libraries.

2.2. Intel MPI

As of R17.2, Intel MPI is now the single MPI type supported for Windows and Linux platforms. It is
bundled in the FENSAP-ICE installation folder (in the mpi/Intel/ subfolder) and will be used auto-
matically from this location. On typical systems, no setup is required.

A different version of Intel MPI can be installed on your system and FENSAP-ICE can be configured to
access it from that location. For advanced options, refer to the official documentation from Intel:

https://software.intel.com/en-us/articles/intel-mpi-library-documentation

2.2.1. Platform Notes

Windows: The default mpiexec option will be set to –localonly. Any change to the mpi arguments
in the FENSAP-ICE Run panel will override this option and use the specified options instead.

Linux: To run on a list of remote hosts, such as on a cluster, a machinefile must be set up in the
mpi argument list. On a queuing system, it might be required to configure it with the following option:
-machinefile $PBS_NODEFILE, where PBS_NODEFILE contains the list of CPU cores.

2.3. MPI Setup in FENSAP-ICE

When MPI is properly configured, launching a multi-CPU run with FENSAP-ICE is straightforward; simply
select the number of CPUs in the Execution setting section of the Start panel.

2.3.1. FENSAP-ICE Advanced MPI Configuration

More flexibility in configuring the execution environment is supported by FENSAP-ICE. Clicking the
Configure button shown above in the Execution settings box opens the following window:

Release 2020 R1 - © ANSYS, Inc. All rights reserved. - Contains proprietary and confidential information
10 of ANSYS, Inc. and its subsidiaries and affiliates.
 MPI Setup in FENSAP-ICE

The usage of the parameters that can be provided to mpirun is shown below.

2.3.1.1. Override mpirun

Select Override mpirun command (default is "mpirun" from $PATH) if the mpirun to be used
is not the default mpirun from the $PATH

2.3.1.2. Additional mpirun Parameters

Additional parameters can be provided to mpirun. For example, the following option overrides the
default settings and permits the usage of a customized list of machines:

-machinefile /path/to/machinefile

In the PBS queuing system, the following option is often required:

-machinefile $PBS_NODEFILE

The mpirun version override and its parameters can be configured with default values by editing
the $NTI_PATH/../config/mpi.txt file. See config/mpi.txt (p. 27).

Release 2020 R1 - © ANSYS, Inc. All rights reserved. - Contains proprietary and confidential information
of ANSYS, Inc. and its subsidiaries and affiliates. 11
MPI

2.3.2. Machinefile
The –machinefile option permits the selection of a customized list of machines for the execution.
The file contains a simple list of machines names on the network (text file). The machines must:

• Share the execution directory via NFS, with identical path on all of them.

• Share access to the same $NTI_PATH directory.

• Have the MPI library installed at the same location.

• (Linux) Be able to communicate with each other using ssh, without password.

• Be able to connect with each other using TCP (firewalls must allow the connection).

• The machine running the master process must be able to access the software licenses (NTI_LI-
CENSE_FILE, ANSYSLMD_LICENSE_FILE).

The machinefile can be tested with the $NTI_PATH/test_mpi command, following the procedure
outlined in the previous chapter.

When using mpirun on the command line, simply add –machinefile filename to the arguments.
The filename argument must have either an absolute or relative path. For example, to use the file:

/home/user/machinefile16

Residing in your home directory, the –machinefile option is:

mpirun –np 16 –machinefile /home/user/machinefile16 $NTI_PATH/test_mpi

The machinefile relative filename can be used when executing in a project subdirectory:

mpirun –np 16 –machinefile ../machinefile16 $NTI_PATH/fensapMPI

2.3.3. Per-Solver machinefile

Runs containing different solvers might have different number of CPUs assigned to them. For example,
in the context of a CHT3D (FENSAP) run, a 64 CPU execution could be configured with 24 CPUs for
C3D, and 16 CPUs for ICE3D:

Release 2020 R1 - © ANSYS, Inc. All rights reserved. - Contains proprietary and confidential information
12 of ANSYS, Inc. and its subsidiaries and affiliates.
 MPI Setup in FENSAP-ICE

The default behavior of most mpirun implementations is to use the machinefiles as-is, in the order
in which the machines are listed. However, this might have the effect of scattering the execution
CPUs over la large number of separate machines.

For example, if a machinefile such as m1,m2,m3,m4,m5,m5,m5,m5 is used for 4 CPUs, the execution
would be scattered in m1, m2, m3, m4.

2.3.4. Saving the Configuration to the Project

The customized MPI settings can be saved for later usage in the same project with the Save button
in the Execution settings panel:

Release 2020 R1 - © ANSYS, Inc. All rights reserved. - Contains proprietary and confidential information
of ANSYS, Inc. and its subsidiaries and affiliates. 13
 Release 2020 R1 - © ANSYS, Inc. All rights reserved. - Contains proprietary and confidential information
14 of ANSYS, Inc. and its subsidiaries and affiliates.
Chapter 3: Queuing Systems
The previous sections illustrated how to configure the machine file and run MPI programs on multiple
computers. Queuing systems usually automate this operation by providing a machinefile tailored to
the requested number of CPUs.

FENSAP-ICE supports the PBS queuing system, or any other queuing system with a compatible syntax.

3.1. PBS

3.1.1. $PBS_NODEFILE
When a job is launched using the PBS queuing system, PBS will provide a list of machines for MPI
execution. This list must be provided to Additional mpirun parameters with the following argument
in the advanced settings:

This option is accessed with the Configure button in the execution window, as shown in the preceding
section. The setting should be saved before proceeding with the execution.

This setting can also be used as a system-wide default in $NTI_PATH/../config/mpi.txt.

3.1.2. PBS Job Settings

In the Configure queue dialog window, some general settings can be specified:

Release 2020 R1 - © ANSYS, Inc. All rights reserved. - Contains proprietary and confidential information
of ANSYS, Inc. and its subsidiaries and affiliates. 15
Queuing Systems

Queue name

This option might be required if there are multiple queues available on the computer. If specified,
the queue name will be translated into the PBS command:

#PBS –q NAME

Nodes ppn

Some queuing systems require the number of processors per node (ppn). For example, on a cluster
composed of quad-core processors, the ppn value would be set to 4. Therefore, to run an 8-CPU job
making efficient use of the resources one must specify two nodes with ppn=4.

#PBS -l nodes=2:ppn=8

If not specified, the ppn default value for PBS is 1. For example:

Example 3.1: Processor per Node

Would request the use of only one processor per node, across eight nodes.

#PBS -l ncpus=8

If PPN is required by the queue, or if the job must run on a specific number of processors on each
node, use the Nodes ppn option.

Example 3.2: 64-processor machine with 8 processors per node

A 32-CPU job on this machine would translate to:

#PBS -l nodes=4:ppn=8

Walltime

Walltime is the maximum time for job execution. It might be mandatory on the PBS system. This
setting will be supplied to PBS in the format:

Release 2020 R1 - © ANSYS, Inc. All rights reserved. - Contains proprietary and confidential information
16 of ANSYS, Inc. and its subsidiaries and affiliates.
 PBS

#PBS -l walltime=24:00:00

Note:

FENSAP-ICE does not expect particular units of time, it simply passes the specified value
to PBS. It is up to you to determine what units of time must be passed to the queue.

3.1.3. The qsub.cfg file

All the PBS settings described here will be written to the qsub.cfg file in the run directory. Therefore,
when executing a FENSAP-ICE module, the file:

$HOME/../Project/run_FENSAP/qsub.cfg will be created. This file is only a PBS wrapper for

the .solvercmd shell script. It may contain, for example:

#!/bin/sh -f

#PBS -N fensap

#PBS -l ncpus=2

#PBS -V

cd /home/user/Project/run_FENSAP/

rm -f fensap.exitStatus fensap.stdout

cp $PBS_NODEFILE pbs_nodes.txt

/home/user/Project/run_FENSAP/.solvercmd 2>&1 > fensap.stdout

echo $? > fensap.exitStatus

The contents of the file can be customized by using the Advanced settings section of the PBS con-
figuration:

Release 2020 R1 - © ANSYS, Inc. All rights reserved. - Contains proprietary and confidential information
of ANSYS, Inc. and its subsidiaries and affiliates. 17
Queuing Systems

PBS header: Commands entered here will be added at the top of the qsub.cfg file.

Job init: Commands entered here will be executed before .solvercmd execution.

Job end: Commands entered here will be executed after .solvercmd execution.

3.1.4. Special Files

When submitting a PBS job, two files are created by FENSAP-ICE for your convenience:

pbs_job.txt

Which contains the job ID, usually used with qstat.

pbs_nodes.txt

Once the job has started, the qsub.cfg execution script will create this file, containing the content
of $PBS_NODEFILE.

FENSAP-ICE will not read these files; they can be removed or modified.

3.2. SGE Queue

SGE is a queuing system similar to PBS and should provide qsub/qstat/qdel commands.

3.2.1. SGE Job Settings

SGE settings are specified in the submittal script via #$ lines, these lines can be configured in FENSAP-
ICE in the Configure… panel of the SGE queue.

Refer to your queuing system documentation, in order to know which parameters to use here.

For example:

Release 2020 R1 - © ANSYS, Inc. All rights reserved. - Contains proprietary and confidential information
18 of ANSYS, Inc. and its subsidiaries and affiliates.
 SGE Queue

Typical parameters:

• Number of CPUs – Required: The command required to specify the number of CPUs will change depending
of the system configuration. FENSAP-ICE will not write the number of CPUs automatically in the submittal
script. The typical syntax for 24 CPUs is:

– #$ -pe mpi 24

• Start in current directory

– #$ -cwd

• Force usage of bash shell

– #$ -S /bin/bash

• Walltime, project identifier, etc.

– Refer to your system documentation.

3.2.2. Special Files

When submitting a PBS job, two files are created by FENSAP-ICE for your convenience:

sge_job.txt

Which contains the job ID, usually used with qstat.

sge_nodes.txt

Once the job is started, the qsub.cfg execution script will create this file, it will be filled with a copy
of the content of the $PE_HOSTFILE.

Release 2020 R1 - © ANSYS, Inc. All rights reserved. - Contains proprietary and confidential information
of ANSYS, Inc. and its subsidiaries and affiliates. 19
Queuing Systems

FENSAP-ICE will not read these files; they can be removed or modified.

3.3. "NULL" Queue

This option will prevent a solver from executing, however all files required for execution will be written
to the run directory, including the script .solvercmd. This script must then be launched manually by
you. The graphical interface will show the execution log and update the convergence graphs until the
Stop → Kill button is pressed.

This queue is useful when the .solvercmd must be edited manually before it is launched, or when
the .solvercmd script is executed on a machine other than the computer where FENSAP-ICE is running.

On Windows, the .solvercmd script must be launched as an argument to nti_sh.exe (provided

in $NTI_PATH).

3.4. "CUSTOM" Queue

Custom queuing commands can be specified using this option; these commands are specified in the
Configure window for the CUSTOM queue. Arguments can be specified on the command lines.

• Start script:

This command is used to start the solver. The start command must launch the .solvercmd. This
command is called when the Start button is pressed.

• Is running script

This command is used to test if the solver process is still running. The command should return a non-
zero exit status if the solver is running; and a zero-exit status if the solver has stopped. (Exit
status: 0=not running 1=running 2=waiting in queue). This command is executed
every few seconds to check the process status. It is not mandatory to specify an Is running script,
however if the command is not specified or fails to execute; FENSAP-ICE will assume the solver is
running until the window is closed. Specify an empty command name if no command is available.

• Stop script

Stops execution (kill .solvercmd execution). This command is called when you select the Kill option
in FENSAP-ICE. This command is also optional; specify an empty command name if no stop command
is available.

• Additional parameters for Start:

The Start command may be given special options supplied by FENSAP-ICE:

– $CPU: Will be substituted by the number of CPU chosen in the Run window.

– $EXEC_DIRECTORY: Substituted by the directory where the .solvercmd is executed (usually, the run
directory).

– $PROJECT_DIRECTORY: Substituted by the project directory, with the entire path.

– $PROJECT_NAME: The project name.

Release 2020 R1 - © ANSYS, Inc. All rights reserved. - Contains proprietary and confidential information
20 of ANSYS, Inc. and its subsidiaries and affiliates.
 "CUSTOM" Queue

– $RUN_DIRECTORY: The run directory.

– $RUN_NAME: The run name (for example Project → run_FENSAP_03 is FENSAP_03)

For example, to replace the PBS queuing system already supported by FENSAP-ICE, do the following:

• start

#!/bin/tcsh

echo running

set number = $argv[1]

@ number = $number / 2

echo "#PBS -l nodes="$number":ppn=2" >sub.cmd

echo "#PBS -N fensap" >>sub.cmd

echo "PBS -V" >>sub.cmd

echo "cd $argv[2]" >>sub.cmd

echo "rm -f fensap.exitStatus fensap.stdout" >>sub.cmd

echo "$argv[2].solvercmd 2>&1 > fensap.stdout" >>sub.cmd

sleep 3

qsub sub.cmd > id_job

• isrunning

#!/bin/tcsh

qstat `cat id_job` > /dev/null

if($status == 0) then

exit 1

else

exit 0

endif

• stop

#!/bin/tcsh

echo stopping job

qdel `cat id_job`

Release 2020 R1 - © ANSYS, Inc. All rights reserved. - Contains proprietary and confidential information
of ANSYS, Inc. and its subsidiaries and affiliates. 21
Queuing Systems

3.4.1. "CUSTOM" Queue for Windows

When FENSAP-ICE launches the .solvercmd script, it uses the nti_sh.exe found in the
$NTI_PATH.

(cd C:\PATH\TO\PROJECT\RUN\)

C:\NTI\bin\nti_sh.exe .solvercmd

A CUSTOM queue needs to execute that command on the target machine. Unlike the UNIX platform,
the CUSTOM queue commands must be executable files, they cannot be shell scripts, since shell
scripts are not a built-in function of Windows. A simple CUSTOM Queue Start command could be:

cmd /C C:\PATH\TO\example.bat

This will execute the windows batch file example.bat The content of this file could be simply:

C:\NTI\bin\nti_sh.exe .solvercmd

Release 2020 R1 - © ANSYS, Inc. All rights reserved. - Contains proprietary and confidential information
22 of ANSYS, Inc. and its subsidiaries and affiliates.
Chapter 4: Advanced MPI Topics
The following sections of this chapter are:
4.1. Software Licenses and Parallel Jobs
4.2. Sweep, Langmuir, FENSAP-ICE-TURBO and Machinefiles

4.1. Software Licenses and Parallel Jobs

The path to the license file (or file server) contained in the environment variable:

(NTI Licensing) $NTI_LICENSE_FILE must be accessible from the machine where the computation
is launched.

(ANSYS Licensing) The licensing environment must be setup in the shell on all machines.

The FENSAP-ICE license contains the maximum number of CPUs that can be used with MPI, and a
number of seats (simultaneous job executions). For example: A single FENSAP license allowing the use
of 16 CPUs can be used in MPI for one 16-CPU job, but cannot be used for two simultaneous 8-CPU
jobs.

In order to run parallel Sweep operations in FENSAP or DROP3D, a sufficient number of software licenses
is required. A 16-CPU DROP3D Langmuir sweep with 4 CPUs per job requires 4 DROP3D licenses (seats).

4.2. Sweep, Langmuir, FENSAP-ICE-TURBO and Machinefiles

In a FENSAP/DROP3D sweep run, or a DROP3D Langmuir run, if the number of CPUs per run is lower
than the total number of CPU allocated for the execution, multiple instances of FENSAP/DROP3D will
run in parallel.

Example 4.1: MPI Settings for 48 CPUs

Release 2020 R1 - © ANSYS, Inc. All rights reserved. - Contains proprietary and confidential information
of ANSYS, Inc. and its subsidiaries and affiliates. 23
Advanced MPI Topics

Example 4.2: Sweep Solver Settings

This example shows that 6 parallel execution of FENSAP will be executed (would require 6 seats of
FENSAP-ICE).

FENSAP-TURBO will also execute multiple parallel jobs. The CPU subdivision among the rows of the
turbomachine is configured in the FENSAP-TURBO Run panel. The number of cores used for each row
depends of the machinefile used.

Multi-core MPI Machine:

If the job is executed on a single machine, with no queuing system nor a machinefile, all the FENSAP
processes will be launched on the same machine. For optimal execution speed, the machine should
have enough CPU-cores to suit the job.

Hyperthreading:

A machine with 8 physical cores and enabled hyperthreading will display as 16 cores in the task manager,
however running with more than 8 CPU-cores on this machine will typically not lead to faster execution
speed.

Machinefile:

If the MPI settings for the job contain the -machinefile parameter, the specified file will be subdivided
to suit the multiple parallel jobs. Some queuing system won’t require the -machinefile argument
in the MPI settings; FENSAP-ICE will then automatically work on $PBS_NODEFILE (PBS) or
$PE_HOSTFILE (SGE).

Example 4.3: With a User-Specified Machinefile

-machinefile /home/username/machines

And the machinefile content:

host1

Release 2020 R1 - © ANSYS, Inc. All rights reserved. - Contains proprietary and confidential information
24 of ANSYS, Inc. and its subsidiaries and affiliates.
 Sweep, Langmuir, FENSAP-ICE-TURBO and Machinefiles

host1

host1

host1

host2

host2

host3

host3

A 8 CPU sweep with 4 CPUs per job would run as:

Job #1: host1 (4 CPUs)

Job #2: host2 (2 CPUs) + host3 (2 CPUs)

At runtime, the machinefile is split in two files nodes_01.txt and nodes_02.txt, which will
be used for each respective mpirun command.

Release 2020 R1 - © ANSYS, Inc. All rights reserved. - Contains proprietary and confidential information
of ANSYS, Inc. and its subsidiaries and affiliates. 25
 Release 2020 R1 - © ANSYS, Inc. All rights reserved. - Contains proprietary and confidential information
26 of ANSYS, Inc. and its subsidiaries and affiliates.
Chapter 5: Configuration Options
The following sections of this chapter are:
5.1. config/gui.txt (Linux only)
5.2. config/mpi.txt
5.3. config/licensing.txt

5.1. config/gui.txt (Linux only)

If present on the system (in the FENSAP-ICE installation directory), this file enables passage of additional
options related to the graphical display.

You can override the global settings in a file in its home directory:

~/.ansys/fensapice/config/gui.txt

Available Options:

MESA=1: Enables launching fensapiceGUI_noGL and nti_3dview_noGL using software rendering.

This is the default option.

MESA=0: Enables launching fensapiceGUI_GL and nti_3dview_GL, using hardware accelerated

rendering. Use on machines with good display drivers, for improved display performance.

Additionally, the behavior can be overridden by launching fensapiceGUI with the -gl or – mesa
command line argument.

5.2. config/mpi.txt
If present on the system (in $NTI_PATH), this file enables loading system-specific default settings for
MPI usage with FENSAP-ICE.

See $NTI_PATH/config/mpi.txt for a description of the available options.

This file should be used to:

• Specify the directory for the MPI version to use, if different from the built-in default.

• Specify machine specific MPI arguments, or arguments required for the queuing system.

• Indicate alternate execution paths on the remote machines (if the remote file-system is different than on
the current machine).

Release 2020 R1 - © ANSYS, Inc. All rights reserved. - Contains proprietary and confidential information
of ANSYS, Inc. and its subsidiaries and affiliates. 27
Configuration Options

5.3. config/licensing.txt
The licensing.txt files enable to force one type of licensing (to disable ANSYS or NTI licensing as
a fallback when the primary licensing check fails).

Variables:

ANSYS_LICENSE_FILE_ONLY=1

Setup to force only ANSYS licensing. The license check will use the global system settings or the config-
uration from the ANSYS Client Licensing Utility.

NTI_LICENSE_FILE=/path/to/license.lic

NTI_LICENSE_FILE=hostname

NTI_LICENSE_FILE=port@host

Setup to force NTI licensing, by specifying directly the hostname of the machine acting as license
server, or the license file containing this information.

Release 2020 R1 - © ANSYS, Inc. All rights reserved. - Contains proprietary and confidential information
28 of ANSYS, Inc. and its subsidiaries and affiliates.
Chapter 6: Directory Structure
The following sections of this chapter are:
6.1. Installation Directory
6.2. User Account – Linux
6.3. User Account - Windows

6.1. Installation Directory

bin/ Contains all the executables:

fensapiceGUI[.exe]: FENSAP-ICE main

graphical environment

fensapMPI[.exe]: FENSAP flow solver (also

used for DROP3D and C3D)

convertgrid[.exe], soln2soln[.exe],
solnEdit[.exe], etc.

Documentation of the command line tools is provided in Command-Line Tools.

data/ Data files required for FENSAP-ICE. The data/

subfolder must be in the same folder as the
bin/ subfolder.
doc/ FENSAP-ICE documentation, in .PDF format.
license/ (NTI Licensing) FLEXlm licensing tools for the
license server. See License Server Setup (p. 1).

lmgrd[.exe]: Server executable

NTI[.exe]: License daemon, must be in the

same directory than lmgrd.exe

lmutil[.exe]: Additional utility commands

LMTOOLS.exe (Windows only): Tool for license

administration.
mpi/ Contains built-in MPI packages.

Intel MPI redistributable files, covered by the

Intel MPI SDK terms.

Release 2020 R1 - © ANSYS, Inc. All rights reserved. - Contains proprietary and confidential information
of ANSYS, Inc. and its subsidiaries and affiliates. 29
Directory Structure

config/ Contains system-wide settings for FENSAP-ICE.

See config/gui.txt (Linux only) (p. 27).

6.2. User Account – Linux

~/.flexlmrc License file configuration file for FLEXLM.
Contains NTI_LICENSE_FILE settings.
~/.ansys/fensapice/ prefs.txt: FENSAP-ICE preference files.

mpi.txt (optional): User override file on the

system-wide mpi.txt defined in the
installation directory.

licensing.txt: Licensing Options

6.3. User Account - Windows

AppData/Roaming/Ansys/fensapice/ prefs.txt: FENSAP-ICE preferences file.

Windows registry

(NTI Licensing) The key NTI_LICENSE_FILE contains the currently selected license server address (or
file location)

Note:

In version 2014R1.0 and older of FENSAP-ICE, preference files were stored in

~/.nti_gui_prefs (Linux)

Appdata/Roaming/NTI/.nti_gui_prefs (Windows)

At first start of FENSAP-ICE 2020 R1, the old files are imported and saved in the new location.

Release 2020 R1 - © ANSYS, Inc. All rights reserved. - Contains proprietary and confidential information
30 of ANSYS, Inc. and its subsidiaries and affiliates.