Doc:Expand and clarify guidance for jvm settings

Change fixed heap size recommendation to a percentage.
Note that large configs may require more stack space

Fixes: elastic#11605 elastic#11842

Fixes elastic#11867

Author: karenzone

docs/static/config-details.asciidoc

@@ -1,20 +1,25 @@
+[[jvm-settings]]
+=== JVM settings
+
+Configure the jvm settings in the `jvm.options` <<settings-files,settings file>>.
+
 [[heap-size]]
-=== Setting the heap size
+==== Setting the JVM heap size
 
-Set the jvm heap size in the `jvm.options` <<config-setting-files,configuration
-file>>.
+Here are some tips for adjusting the JVM heap size:
 
-NOTE: The recommended heap size for typical ingestion scenarios should be no
+// tag::heap-size-tips[]
+* The recommended heap size for typical ingestion scenarios should be no
 less than 4GB and no more than 8GB.
 
-Here are some additional tips for adjusting the JVM heap size:
-
 * CPU utilization can increase unnecessarily if the heap size is too low,
 resulting in the JVM constantly garbage collecting. You can check for this issue
 by doubling the heap size to see if performance improves. 
 
-* Do not increase the heap size past the amount of physical
-memory. Leave at least 1GB free for the OS and other processes.
+* Do not increase the heap size past the amount of physical memory. Some memory
+must be left to run the OS and other processes.  As a general guideline for most
+installations, don't exceed 50-75% of physical memory. The more memory you have,
+the higher percentage you can use.
 
 * Set the minimum (Xms) and maximum (Xmx) heap allocation size to the same
 value to prevent the heap from resizing at runtime, which is a very costly
@@ -23,12 +28,31 @@ process.
 * You can make more accurate measurements of the JVM heap by using either the
 `jmap` command line utility distributed with Java or by using VisualVM. For more
 info, see <<profiling-the-heap>>.
+// end::heap-size-tips[]
 
+[[stacks-size]]
+==== Setting the JVM stack size
 
+Large configurations may require additional JVM stack memory.
+If you see a stack overflow error, try increasing the JVM stack size. 
+Add an entry similar to this one in the `jvm.options`
+<<settings-files,settings file>>:
 
+[source,sh]
+-----
+-Xss4M 
+-----
 
+Note that the default stack size is different per platform and per OS
+flavor. You can find out what the default is by running:
 
+[source,sh]
+-----
+java -XX:+PrintFlagsFinal -version | grep ThreadStackSize
+-----
 
+Depending on the default stack size, start by multiplying by 4x, then 8x, and
+then 16x until the overflow error resolves.
 
 
 

docs/static/performance-checklist.asciidoc

@@ -8,11 +8,18 @@ performance:
 * <<tuning-logstash>>
 
 [[performance-troubleshooting]]
-=== Performance Troubleshooting Guide
+=== Performance Troubleshooting
 
-You can use this troubleshooting guide to quickly diagnose and resolve Logstash performance problems. Advanced knowledge of pipeline internals is not required to understand this guide. However, the <<pipeline,pipeline documentation>> is recommended reading if you want to go beyond this guide.
+You can use these troubleshooting tips to quickly diagnose and resolve Logstash performance problems. 
+Advanced knowledge of pipeline internals is not required to understand this guide. 
+However, the <<pipeline,pipeline documentation>> is recommended reading if you want to go beyond these tips.
 
-You may be tempted to jump ahead and change settings like `pipeline.workers` (`-w`) as a first attempt to improve performance. In our experience, changing this setting makes it more difficult to troubleshoot performance problems because you increase the number of variables in play. Instead, make one change at a time and measure the results. Starting at the end of this list is a sure-fire way to create a confusing situation.
+You may be tempted to jump ahead and change settings like `pipeline.workers`
+(`-w`) as a first attempt to improve performance. In our experience, changing
+this setting makes it more difficult to troubleshoot performance problems
+because you increase the number of variables in play. Instead, make one change
+at a time and measure the results. Starting at the end of this list is a
+sure-fire way to create a confusing situation.
 
 [float]
 ==== Performance Checklist
@@ -40,10 +47,7 @@ You may be tempted to jump ahead and change settings like `pipeline.workers` (`-
 
 . *Check the JVM heap:*
 +
-* Often times CPU utilization can go through the roof if the heap size is too low, resulting in the JVM constantly garbage collecting.
-* A quick way to check for this issue is to double the heap size and see if performance improves. Do not increase the heap size past the amount of physical memory. Leave at least 1GB free for the OS and other processes.
-* You can make more accurate measurements of the JVM heap by using either the `jmap` command line utility distributed with Java or by using VisualVM. For more info, see <<profiling-the-heap>>.
-* Always make sure to set the minimum (Xms) and maximum (Xmx) heap allocation size to the same value to prevent the heap from resizing at runtime, which is a very costly process.
+include::config-details.asciidoc[tag=heap-size-tips]
 
 . *Tune Logstash worker settings:*
 +

docs/static/setting-up-logstash.asciidoc

@@ -168,6 +168,7 @@ to standard output.
 Logstash has two types of configuration files: _pipeline configuration files_, which define the Logstash processing
 pipeline, and _settings files_, which specify options that control Logstash startup and execution.
 
+[[pipeline-config-files]]
 ==== Pipeline Configuration Files
 
 You create pipeline configuration files when you define the stages of your Logstash processing pipeline. On deb and
@@ -176,6 +177,7 @@ files with `.conf` extension in the `/etc/logstash/conf.d directory` and ignores
 
 See <<configuration>> for more info.
 
+[[settings-files]]
 ==== Settings Files
 
 The settings files are already defined in the Logstash installation. Logstash includes the following settings files: