Merge pull request echocat#258 from echocat/develop

backmerge for release 6.0.0

Author: dwerder

CHANGELOG.md

@@ -1,3 +1,23 @@
+## 2016-03-30 - 6.0.0 (Major release)
+
+#### Features:
+
+- Nginx/Gunicorn is now fully supported on both Debian and RedHat-like systems
+- The correct username/group is now used across apache/nginx for both Debian and RedHat
+- Pip is now the default provider for Django to ensure mutually-working package versions are installed.
+- Added systemd scripts for when `$service_provider == 'systemd'`
+- Added workaround for [race condition bug in graphite](graphite-project/graphite-web#403)
+- Added configuration variable `gr_web_server_remove_default` -- default config is only wiped when the variable is true, or when it is `undef` and `gr_web_server_port` == 80
+- Renamed `gr_apache_port` and `gr_apache_port_https` to `gr_web_server_port` and `gr_web_server_port` respectively. If the old configuration values are used, puppet fails telling you to use the new ones.
+- Add param for carbon-relay init.d ulimit `gr_carbon_relay_ulimit`
+- Add support for custom directory locations , see `gr_base_dir` and `gr_*_dir`
+- Add `gr_relay_diverse_replicas` option, default to true
+
+#### Bugfixes:
+
+- python-crypto is obsoleted on RHEL7, replaced by python2-crypto
+- Redhat: avoid raise condition on initd restart
+
 ## 2016-01-29 - 5.16.1 (Bugfix release)
 
 #### Bugfixes:

Gemfile

@@ -7,3 +7,5 @@ gem 'puppetlabs_spec_helper', '>= 0.1.0'
 gem 'puppet-lint', '>= 0.3.2'
 gem 'facter', facterversion
 gem 'rspec', '< 3.2.0'
+# rubi <1.9 versus rake 11.0.0 workaround
+gem 'rake', '< 11.0.0'

README.md

@@ -21,6 +21,10 @@
 
 This module installs and makes basic configs for graphite, with carbon and whisper.
 
+[![Puppet Forge](http://img.shields.io/puppetforge/v/dwerder/graphite.svg)](https://forge.puppetlabs.com/dwerder/graphite)
+[![Build Status](https://secure.travis-ci.org/echocat/puppet-graphite.png?branch=master)](https://travis-ci.org/echocat/puppet-graphite)
+[![Puppet Forge Downloads](http://img.shields.io/puppetforge/dt/dwerder/graphite.svg)](https://forge.puppetlabs.com/dwerder/graphite)
+
 ##Module Description
 
 [Graphite](http://graphite.readthedocs.org/en/latest/overview.html), and its components Carbon and Whisper, is an enterprise-scale monitoring tool. This module sets up a simple graphite server with all its components. Furthermore it can be used to set up more complex graphite environments with metric aggregation, clustering and so on.
@@ -195,9 +199,9 @@ ones set for the principal instance.
 ```puppet
    class {'graphite':
       gr_line_receiver_port => 2003,
-      gr_pickle_receiver_port => 2004, 
+      gr_pickle_receiver_port => 2004,
       gr_cache_query_port => 7002,
-      
+
       gr_cache_instances => {
          'cache:b' => {
             'LINE_RECEIVER_PORT' => 2103,
@@ -216,8 +220,8 @@ ones set for the principal instance.
 So in this case you would have 3 cache instances, the first one is `cache` (you can refer to it as `cache:a` too), `cache:b` and `cache:c`. cache:a will listen on ports 2003, 2004 and 7002 for line, pickle and query respectively. But, cache:b will do it on ports 2103, 2104, and 7102, and cache:c on 2203, 2204 and 7202. All other parameters from cache:a will be inherited by cache:b and c.
 
 ###Installing with something other than pip and specifying package names and versions
-If you need to install via something other pip, an internal apt repo with fpm converted packages for instance, you can set `gr_pip_install` to false.
-If you're doing this you'll most likely have to override the default package names and versions as well. 
+If you need to install via something other than pip, an internal apt repo with fpm converted packages for instance, you can set `gr_pip_install` to false.
+If you're doing this you'll most likely have to override the default package names and versions as well.
 ```puppet
   class { '::graphite':
     gr_pip_install        => false,
@@ -298,6 +302,11 @@ The prefix to be applied to internal performance metrics. Defaults to 'carbon'.
 
 Default is 60. Set the interval between sending internal performance metrics; affects all carbon daemons.
 
+#####`gr_carbon_relay_ulimit`
+
+Default is undef. Set the maximum number of file descriptors for
+carbon-relay process.
+
 #####`gr_line_receiver_interface`
 
 Default is '0.0.0.0' (string). Interface the line receiver listens.
@@ -358,13 +367,54 @@ Default is 7002. Self explaining.
 
 Default is 'GMT' (string). Timezone for graphite to be used.
 
+#####`gr_base_dir`
+
+Default is '/opt/graphite'. Set base install location of Graphite. This forms the base location for installs, predominantly appropriate for pip installations. When not installing using pip a typical location for this may be '/opt/carbon'.
+
+#####`gr_storage_dir`
+
+Default is '${gr_base_dir}/storage'. Set location of base storage files.  When not installing using pip a typical location for this may be '/opt/carbon'.
+
 #####`gr_local_data_dir`
 
-Default is '/opt/graphite/storage/whisper'. Set location of whisper files.
+Default is '${gr_storage_dir}/whisper'. Set location of whisper files.
 
 #####`gr_rrd_dir`
 
-Default is '/opt/graphite/storage/rrd'. Set location of rrd data files.
+Default is '${gr_storage_dir}/rrd'. Set location of rrd data files.
+
+#####`gr_whitelists_dir`
+
+Default is '${gr_storage_dir}/rrd'. Set location of whitelist configuration files.
+
+#####`gr_carbon_conf_dir`
+
+Default is '${gr_base_dir}/conf'. Set location of Carbon's configuration files. Most relevant when not using pip for installation. A typical location for this may be '/etc/carbon'.
+
+#####`gr_carbon_log_dir`
+
+Default is '${gr_storage_dir}/log/carbon-cache'. Set location of carbon cache log files.
+
+#####`gr_graphiteweb_log_dir`
+
+Default is '${gr_storage_dir}/log'. Set location of graphite web log files.
+
+#####`gr_graphiteweb_conf_dir`
+
+Default is '${gr_base_dir}/conf'. Set location of graphite web configuration.
+
+#####`gr_graphiteweb_webapp_dir`
+
+Default is '${gr_base_dir}/webapp'. Set location of graphite web's webapp files.
+
+#####`gr_graphiteweb_storage_dir`
+
+Default is '/var/lib/graphite-web'. Set location of graphite web's storage, used for graphite.db file.
+
+#####`gr_graphiteweb_install_lib_dir`
+
+Default is '${gr_graphiteweb_webapp_dir}/graphite'. Set location of libraries directory for graphite web.
+
 
 #####`gr_storage_schemas`
 
@@ -400,7 +450,19 @@ The storage aggregation rules.
 
 #####`gr_web_server`
 
-Default is 'apache'. The web server to use. Valid values are 'apache', 'nginx', 'wsgionly' or 'none'. 'nginx' is only supported on Debian-like systems. And 'none' means that you will manage the webserver yourself.
+Default is 'apache'. The web server to configure. Valid values are 'apache', 'nginx', 'wsgionly' or 'none'.
+
+Apache is configured with mod_wsgi, nginx is configured with gunicorn. 'wsgionly' configures only gunicorn.
+
+The value 'none' means that you will manage the webserver yourself.
+
+#####`gr_web_server_port`
+
+Default is 80. The HTTP port which the web server will use. Only used for $gr_web_server => 'apache' or 'nginx'.
+
+#####`gr_web_server_port_https`
+
+Default is 443. The HTTPS port which the web server will use. Only used for $gr_web_server => 'apache'.
 
 #####`gr_web_servername`
 
@@ -429,19 +491,11 @@ Path to SSL dir containing keys and certs. Default is undef.
 
 #####`gr_web_group`
 
-Default is undef. Group name to chgrp the files that will served by webserver.  Use only with gr_web_server => 'wsgionly' or 'none'.
+Group name to chgrp the files that will served by webserver. Only necessary for gr_web_server => 'wsgionly' or 'none'.
 
 #####`gr_web_user`
 
-Default is undef. Username to chown the files that will served by webserver.  Use only with gr_web_server => 'wsgionly' or 'none'.
-
-#####`gr_apache_port`
-
-Default is 80. The HTTP port apache will use.
-
-#####`gr_apache_port_https`
-
-Default is 443. The HTTPS port apache will use.
+Username to chown the files that will served by webserver. Only necessary for gr_web_server => 'wsgionly' or 'none'.
 
 #####`gr_apache_conf_template`
 
@@ -537,6 +591,10 @@ Default is 'rules'
 
 Default is 1 (integer). Add redundancy by replicating every datapoint to more than one machine.
 
+#####`gr_relay_diverse_replicas`
+
+Default is 'True' (string). Add to guarantee replicas across distributed hosts.
+
 #####`gr_relay_destinations`
 
 Default  is [ '127.0.0.1:2004' ] (array). Array of backend carbons for relay.
@@ -724,7 +782,7 @@ Default is 30.  value to pass to gunicorns --timeout arg.
 Default is 'unix:/var/run/graphite.sock'.  value to pass to gunicorns --bind arg.
 
 #####`gunicorn_workers`
-  
+
 Default is 2. value to pass to gunicorn's --worker arg.
 
 #####`gr_cache_instances`    
@@ -865,7 +923,7 @@ On Redhat distributions you need the EPEL or RPMforge repository, because Graphi
 
 ##Limitations
 
-This module is tested on CentOS 6.5 and Debian 7 (Wheezy) and should also run without problems on
+This module is tested on CentOS 6.5 and Debian 7 (Wheezy) and should also run on
 
 * RHEL/CentOS/Scientific 6+
 * Debian 6+
@@ -874,8 +932,17 @@ This module is tested on CentOS 6.5 and Debian 7 (Wheezy) and should also run wi
 Most settings of Graphite can be set by parameters. So their can be special configurations for you. In this case you should edit
 the file `templates/opt/graphite/webapp/graphite/local_settings.py.erb`.
 
-The nginx configs are only supported on Debian based systems at the moment.
+###Compatibility Notes
+* There is currently an [open ticket](https://tickets.puppetlabs.com/browse/PUP-3829) with Puppet about broken pip support in CentOS 6/7. The
+workaround for this bug is to create a symlink from `/usr/bin/pip-python` (which doesn't exist) to `/usr/bin/pip` (which does).
+* CentOS 7's default `nginx.conf` includes a `server` section listening on port 80. Thus, it is not possible to set up graphite without modifying
+the package-provided configuration file. You will have to either manually remove the `server` section, or provide a `gr_web_server_port` other
+than port 80.
+* nginx/gunicorn requires a `systemctl restart gunicorn` after installing on Ubuntu 15.10
+* SELinux must be disabled
 
 ##Contributing
 
 Echocat modules are open projects. So if you want to make this module even better, you can contribute to this module on [Github](https://github.com/echocat/puppet-graphite).
+
+Make sure to read the repository's `DEVELOP.md` file first.

files/fix-graphite-race-condition.py

@@ -0,0 +1,18 @@
+import sys
+sys.path.append('/opt/graphite/webapp')
+from django.contrib.auth.models import User
+from graphite.account.models import Profile
+from graphite.logger import log
+try:
+  defaultUser = User.objects.get(username='default')
+except User.DoesNotExist:
+  randomPassword = User.objects.make_random_password(length=16)
+  defaultUser = User.objects.create_user('default','[email protected]',randomPassword)
+  defaultUser.save()
+
+try:
+  defaultProfile = Profile.objects.get(user=defaultUser)
+except Profile.DoesNotExist:
+  defaultProfile = Profile(user=defaultUser)
+  defaultProfile.save()
+