Apache

Installation
RHEL / CentOS

Apache can be installed via yum as shown below:
[root@web01 ~]# yum install httpd httpd-tools mod_ssl

When you setup Apache, its common to also setup PHP. The customer will be able to tell you what version of PHP they want, and also what PHP modules they need. Below are some examples for the more common requests:

PHP 7.1 (IUS)
[root@web01 ~]# yum install php71u php71u-gd php71u-mysqlnd php71u-opcache php71u-xml php71u-devel

PHP 7.2 (IUS)
[root@web01 ~]# yum install php72u php72u-gd php72u-mysqlnd php72u-opcache php72u-xml php72u-devel

PHP 7.3 (IUS)
[root@web01 ~]# yum install mod_php73 php73-gd php73-mysqlnd php73-opcache php73-xml php73-devel

Distribution default for RHEL/CentOS 6: PHP 5.3
[root@web01 ~]# yum install php php-gd php-mysql php-xml php-devel

Distribution default for RHEL/CentOS 7: PHP 5.4
[root@web01 ~]# yum install php php-gd php-mysql php-xml php-devel

## Distribution default for RHEL/CentOS 8: PHP 7.2
[root@web01 ~]# yum install php php-gd php-mysqlnd php-xml php-devel

Now setup directories to store site configuration and content:

[root@web01 ~]# mkdir -p /var/www/vhosts
[root@web01 ~]# mkdir -p /etc/httpd/vhost.d

Configure Apache to load vhosts from the vhost.d directory:

RHEL/CentOS 6
[root@web01 ~]# echo “Include vhost.d/*.conf” >> /etc/httpd/conf/httpd.conf

RHEL/CentOS 7 & 8
[root@web01 ~]# echo “IncludeOptional vhost.d/*.conf” >> /etc/httpd/conf/httpd.conf

Startup Apache and set it to start on boot

RHEL/CentOS 6
[root@web01 ~]# chkconfig httpd on
[root@web01 ~]# service httpd start

RHEL/CentOS 7 & 8
[root@web01 ~]# systemctl enable httpd
[root@web01 ~]# systemctl start httpd

Ubuntu / Debian

Install Apache and PHP for your distribution accordingly:

Ubuntu 14.04
[root@web01 ~]# apt-get update
[root@web01 ~]# apt-get install libapache2-mod-php5 apache2-utils php5-cli php-pear php5-mysql php-apc php5
gd php5-dev php5-curl php5-mcrypt

Ubuntu 16.04
[root@web01 ~]# apt-get update
[root@web01 ~]# apt-get install libapache2-mod-php libapache2-mod-php7.0 apache2 apache2-utils php7.0-cli
php-pear php7.0-mysql php7.0-gd php7.0-dev php7.0-curl php7.0-mcrypt php7.0-opcache

Ubuntu 18.04
[root@web01 ~]# apt update
[root@web01 ~]# apt install libapache2-mod-php libapache2-mod-php apache2 apache2-utils php-cli php-pear php
mysql php-gd php-dev php-curl php-opcache

Ubuntu 20.04
[root@web01 ~]# apt update
[root@web01 ~]# apt install libapache2-mod-php libapache2-mod-php apache2 apache2-utils php-cli php-pear php
mysql php-gd php-dev php-curl php-opcache
Enable and disable a few modules so everything works:

Ubuntu 14.04
[root@web01 ~]# /usr/sbin/a2enmod access_compat alias auth_basic authn_core authn_file authz_core
authz_groupfile authz_host authz_user autoindex deflate dir env filter mime mpm_prefork negotiation rewrite
setenvif socache_shmcb ssl status php5
[root@web01 ~]# /usr/sbin/php5enmod mcrypt

Ubuntu 16.04
[root@web01 ~]# /usr/bin/a2dismod mpm_event
[root@web01 ~]# /usr/sbin/a2enmod access_compat alias auth_basic authn_core authn_file authz_core
authz_groupfile authz_host authz_user autoindex deflate dir env filter mime mpm_prefork negotiation rewrite
setenvif socache_shmcb ssl status php7.0 mpm_prefork
[root@web01 ~]# /usr/sbin/phpenmod mcrypt opcache

Ubuntu 18.04
[root@web01 ~]# /usr/sbin/a2dismod mpm_event
[root@web01 ~]# /usr/sbin/a2enmod access_compat alias auth_basic authn_core authn_file authz_core
authz_groupfile authz_host authz_user autoindex deflate dir env filter mime mpm_prefork negotiation rewrite
setenvif socache_shmcb ssl status php7.2 mpm_prefork
[root@web01 ~]# /usr/sbin/phpenmod opcache

Ubuntu 20.04
[root@web01 ~]# /usr/sbin/a2dismod mpm_event
[root@web01 ~]# /usr/sbin/a2enmod access_compat alias auth_basic authn_core authn_file authz_core
authz_groupfile authz_host authz_user autoindex deflate dir env filter mime mpm_prefork negotiation rewrite
setenvif socache_shmcb ssl status php7.4 mpm_prefork
[root@web01 ~]# /usr/sbin/phpenmod opcache
Now setup directories to store site content:
[root@web01 ~]# mkdir -p /var/www/vhosts
Startup Apache and set it to start on boot

Ubuntu 14.04
[root@web01 ~]# service apache2 restart
## Ubuntu 16.04, 18.04 and 20.04
[root@web01 ~]# systemctl enable apache2
[root@web01 ~]# systemctl restart apache2

Patching

Assuming the package was installed via a package manager, updates can be performed by running the commands below:

RHEL / CentOS
[root@web01 ~]# yum update httpd

Debian / Ubuntu
[root@web01 ~]# apt-get update
[root@web01 ~]# apt-get install apache2

If the package was installed from source, the vendors documentation will need to be consulted.

Configuration

Adding new sites
Setting up additional websites in Apache is easy once you have a good starting point. Some information that will need to be known beforehand is below:

• Domain name
• SFTP username
• SFTP password

For the purposes of this guide, the domain will be example.com, and the SFTP username will be example.

First, create the directory that will be storing your new website content:

[root@web01 ~]# mkdir -p /var/www/vhosts/example.com

Then create the SFTP username and password
[root@web01 ~]# useradd -d /var/www/vhosts/example.com example
[root@web01 ~]# passwd example
[root@web01 ~]# chown example:example /var/www/vhosts/example.com
Now create the Apache Virtualhost configuration. Below are two generic configurations for the desired operating system:
## RHEL / CentOS
[root@web01 ~]# vim /etc/httpd/vhost.d/example.com.conf
<VirtualHost *:80>
ServerName example.com
ServerAlias www.example.com
DocumentRoot /var/www/vhosts/example.com

# Force HTTPS when loading the page
#RewriteEngine On
#RewriteCond %{HTTPS} off
#RewriteRule (.*) https://%{HTTP_HOST}%{REQUEST_URI}

<Directory /var/www/vhosts/example.com>
Options -Indexes +FollowSymLinks -MultiViews
AllowOverride All
</Directory>

CustomLog /var/log/httpd/example.com-access.log combined
ErrorLog /var/log/httpd/example.com-error.log

# Possible values include: debug, info, notice, warn, error, crit,
# alert, emerg.
LogLevel warn
</VirtualHost>

#<VirtualHost *:443>
# ServerName example.com
# ServerAlias www.example.com
# DocumentRoot /var/www/vhosts/example.com
# <Directory /var/www/vhosts/example.com>
# Options -Indexes +FollowSymLinks -MultiViews
# AllowOverride All
# </Directory>
#
# CustomLog /var/log/httpd/example.com-access.log combined
# ErrorLog /var/log/httpd/example.com-error.log
#
# # Possible values include: debug, info, notice, warn, error, crit,
# # alert, emerg.
# LogLevel warn
#
# SSLEngine on
# SSLCertificateKeyFile /etc/pki/tls/private/2019-example.com.key
# SSLCertificateFile /etc/pki/tls/certs/2019-example.com.crt
# SSLCertificateChainFile /etc/pki/tls/certs/2019-example.com.ca.crt
#
# <FilesMatch “\.(cgi|shtml|phtml|php)$”>
# SSLOptions +StdEnvVars
# </FilesMatch>
#
# BrowserMatch “MSIE [2-6]” nokeepalive ssl-unclean-shutdown
downgrade-1.0 force-response-1.0
# BrowserMatch “MSIE [17-9]” ssl-unclean-shutdown
#</VirtualHost>

Ubuntu / Debian
[root@web01 ~]# vim /etc/httpd/sites-available/example.com.conf
<VirtualHost *:80>
ServerName example.com
ServerAlias www.example.com
DocumentRoot /var/www/vhosts/example.com
# Force HTTPS when loading the page
#RewriteEngine On
#RewriteCond %{HTTPS} off
#RewriteRule (.*) https://%{HTTP_HOST}%{REQUEST_URI}
<Directory /var/www/vhosts/example.com>
Options -Indexes +FollowSymLinks -MultiViews
AllowOverride All
</Directory>

CustomLog /var/log/apache2/example.com-access.log combined
ErrorLog /var/log/apache2/example.com-error.log
# Possible values include: debug, info, notice, warn, error, crit,
# alert, emerg.
LogLevel warn
</VirtualHost>

#<VirtualHost *:443>
# ServerName example.com
# ServerAlias www.example.com
# DocumentRoot /var/www/vhosts/example.com
# <Directory /var/www/vhosts/example.com>
# Options -Indexes +FollowSymLinks -MultiViews
# AllowOverride All
# </Directory>
#
# CustomLog /var/log/apache2/example.com-access.log combined
# ErrorLog /var/log/apache2/example.com-error.log
#
# # Possible values include: debug, info, notice, warn, error, crit,
# # alert, emerg.
# LogLevel warn
#
# SSLEngine on
# SSLCertificateKeyFile /etc/pki/tls/private/2019-example.com.key
# SSLCertificateFile /etc/pki/tls/certs/2019-example.com.crt
# SSLCertificateChainFile /etc/pki/tls/certs/2019-example.com.ca.crt
#
# <FilesMatch “\.(cgi|shtml|phtml|php)$”>
# SSLOptions +StdEnvVars
# </FilesMatch>
#
# BrowserMatch “MSIE [2-6]” nokeepalive ssl-unclean-shutdown
downgrade-1.0 force-response-1.0
# BrowserMatch “MSIE [17-9]” ssl-unclean-shutdown
#</VirtualHost>

Test the configuration and restart Apache so the new changes go into effect:
## RHEL / CentOS
[root@web01 ~]# apachectl configtest
[root@web01 ~]# service httpd restart

Ubuntu / Debian
[root@web01 ~]# apache2ctl configtest
[root@web01 ~]# service apache2 restart

Finally, confirm the new site is actually working in Apache:

[root@web01 ~]# echo “Test page for example.com” > /var/www/vhosts/example.com/rstest.html
[root@web01 ~]# curl -H “Host: www.example.com” localhost/rstest.html
Test page for example.com
[root@web01 ~]# rm /var/www/vhosts/example.com/rstest.html

If you see something other than what your test file was, then troubleshoot things accordingly. Otherwise, update the ticket letting the customer know that their new domain is setup and how they can access it via SFTP. An example setup letter is below:

Good afternoon,
As requested, your new domain has been created. The information you will need to access and use your new domain is posted below
Website Information

Domain: example.com
DocumentRoot: /var/www/vhosts/example.com
SFTP IP: 123.123.123.123
SFTP Username: example
SFTP Password: ubersecurepassword

DNS Information

Once you are ready to go live, you will want to point the DNS as follows:
www.example.com IN A 123.123.123.123
example.com. IN A 123.123.123.123

If you want to be able to test the domain before you update DNS, I’ll provide a link below that walks you
through modifying the hosts file on your workstation:
https://support.rackspace.com/how-to/modify-your-hosts-file/

The entry you want to put in the hosts file is:
123.123.123.123 www.example.com example.com

Regards,
[YOUR_SIGNATURE_HERE]

Adding or updating SSL Certificates

A common request is to install a new SSL certificate for a website. There are 4 things you need from the customer before you can begin:

• The domain in question
• The private key
• The certificate
• The CA certificate (Not always required, depends on the SSL provider)

First, confirm the new private key matches the new certificate by checking the modulus . If the output of the two commands are not identical, then the

[root@web01 ~]# openssl rsa -noout -modulus -in example.com.key | openssl md5
(stdin)= 631e818f6cc249fca465c23c3a070b35
[root@web01 ~]# openssl x509 -noout -modulus -in example.com.crt | openssl md5
(stdin)= 631e818f6cc249fca465c23c3a070b35

Now check to see if there were any SSL certificates already setup for the domain. If so, back them up by:

Check for the presence of existing SSL certificates
[root@web01 ~]# cat /etc/httpd/vhost.d/example.com.conf |grep SSL
SSLCertificateKeyFile /etc/pki/tls/private/example.com.key
SSLCertificateFile /etc/pki/tls/certs/example.com.crt
SSLCertificateChainFile /etc/pki/tls/certs/example.com.ca.crt

Backup the existing SSL certificates
[root@web01 ~]# mv /etc/pki/tls/private/example.com.key /etc/pki/tls/private/2018-example.com.key
[root@web01 ~]# mv /etc/pki/tls/certs/example.com.crt /etc/pki/tls/certs/2018-example.com.crt
[root@web01 ~]# mv /etc/pki/tls/certs/example.com.ca.crt /etc/pki/tls/certs/2018-example.com.ca.crt

Then copy/paste the new SSL certificates to the following files accordingly, changing the year and domain accordingly:

• The private key : /etc/pki/tls/private/2019-example.com.key
• The certificate : /etc/pki/tls/certs/2019-example.com.crt
• The CA certificate : /etc/pki/tls/certs/2019-example.com.ca.crt

Now update the domains vhost configuration to point to the SSL certificates to the new files you just created. The examples below contain the full output of the vhost file in case the 443 block needs to be added:

RHEL / CentOS
[root@web01 ~]# vim /etc/httpd/vhost.d/example.com.conf
<VirtualHost *:80>
ServerName example.com
ServerAlias www.example.com
DocumentRoot /var/www/vhosts/example.com
# Force HTTPS when loading the page
#RewriteEngine On
#RewriteCond %{HTTPS} off
#RewriteRule (.*) https://%{HTTP_HOST}%{REQUEST_URI}
<Directory /var/www/vhosts/example.com>
Options -Indexes +FollowSymLinks -MultiViews
AllowOverride All
</Directory>
CustomLog /var/log/httpd/example.com-access.log combined
ErrorLog /var/log/httpd/example.com-error.log
# Possible values include: debug, info, notice, warn, error, crit,
# alert, emerg.
LogLevel warn
</VirtualHost>

<VirtualHost *:443>
ServerName example.com
ServerAlias www.example.com
DocumentRoot /var/www/vhosts/example.com
<Directory /var/www/vhosts/example.com>
Options -Indexes +FollowSymLinks -MultiViews
AllowOverride All
</Directory>
CustomLog /var/log/httpd/example.com-access.log combined
ErrorLog /var/log/httpd/example.com-error.log
# Possible values include: debug, info, notice, warn, error, crit,
# alert, emerg.
LogLevel warn
SSLEngine on
SSLCertificateKeyFile /etc/pki/tls/private/2019-example.com.key
SSLCertificateFile /etc/pki/tls/certs/2019-example.com.crt
SSLCertificateChainFile /etc/pki/tls/certs/2019-example.com.ca.crt
<FilesMatch “\.(cgi|shtml|phtml|php)$”>
SSLOptions +StdEnvVars
</FilesMatch>
BrowserMatch “MSIE [2-6]” nokeepalive ssl-unclean-shutdown downgrade
1.0 force-response-1.0
BrowserMatch “MSIE [17-9]” ssl-unclean-shutdown
</VirtualHost>

Ubuntu / Debian
[root@web01 ~]# vim /etc/httpd/sites-available/example.com.conf
<VirtualHost *:80>
ServerName example.com
ServerAlias www.example.com
DocumentRoot /var/www/vhosts/example.com
# Force HTTPS when loading the page
#RewriteEngine On
#RewriteCond %{HTTPS} off
#RewriteRule (.*) https://%{HTTP_HOST}%{REQUEST_URI}
<Directory /var/www/vhosts/example.com>
Options -Indexes +FollowSymLinks -MultiViews
AllowOverride All
</Directory>
CustomLog /var/log/apache2/example.com-access.log combined
ErrorLog /var/log/apache2/example.com-error.log
# Possible values include: debug, info, notice, warn, error, crit,
# alert, emerg.
LogLevel warn
</VirtualHost>

<VirtualHost *:443>
ServerName example.com
ServerAlias www.example.com
DocumentRoot /var/www/vhosts/example.com
<Directory /var/www/vhosts/example.com>
Options -Indexes +FollowSymLinks -MultiViews
AllowOverride All
</Directory>
CustomLog /var/log/apache2/example.com-access.log combined
ErrorLog /var/log/apache2/example.com-error.log
# Possible values include: debug, info, notice, warn, error, crit,
# alert, emerg.
LogLevel warn
SSLEngine on
SSLCertificateKeyFile /etc/pki/tls/private/2019-example.com.key
SSLCertificateFile /etc/pki/tls/certs/2019-example.com.crt
SSLCertificateChainFile /etc/pki/tls/certs/2019-example.com.ca.crt
<FilesMatch “\.(cgi|shtml|phtml|php)$”>
SSLOptions +StdEnvVars
</FilesMatch>
BrowserMatch “MSIE [2-6]” nokeepalive ssl-unclean-shutdown downgrade
1.0 force-response-1.0
BrowserMatch “MSIE [17-9]” ssl-unclean-shutdown
</VirtualHost>

Secure the permissions on the new SSL key since the system will set it to 644 by default which is insecure:

[root@web01 ~]# chmod 600 /etc/pki/tls/private/2019-example.com.key

Now test the Apache configuration to ensure there are no errors, then restart Apache by running:

RHEL / CentOS
[root@web01 ~]# apachectl configtest
[root@web01 ~]# service httpd restart

Ubuntu / Debian
[root@web01 ~]# apache2ctl configtest
[root@web01 ~]# service apache2 restart

Finally, test the domain in your browser by going to https://www.example.com, then confirm the SSL certificate is valid, and the expiration is set for some point far in the future.

Configuring Apache mod_status

The Apache mod_status module can be very useful when troubleshooting high CPU or memory usage within Apache. The mod_status module provides details such as:

• The number of workers serving requests
• The number of idle workers
• The status of each worker, the number of requests that worker has performed and the total number of bytes served by the worker.
• A total number of accesses and byte count served.
• The time the server was started/restarted and the time it has been running for.
• Averages giving the number of requests per second, the number of bytes served per second and the average number of bytes per request.
• The current percentage CPU used by each worker and in total by all workers combined.
• The current hosts and requests being processed.

Enable mod_status is simple. Though it can be a bit difficult to explain in a guide as the setup is slightly different depending on what operating system you are using. The file that needs to be updated for each operating system is shown below:

RHEL / CentOS
[root@web01 ~]# vim /etc/httpd/conf.d/status.conf

Ubuntu 14.04
[root@web01 ~]# vim /etc/apache2/conf-available/status.conf

Ubuntu 16.04, 18.04 and 20.04
[root@web01 ~]# vim /etc/apache2/mods-available/status.conf

Using the file location for your operating system shown above, use the following configuration to enable mod_status. Be sure to update the
AuthUserFile line accordingly for your operating system:

<IfModule mod_status.c>
#
# ExtendedStatus controls whether Apache will generate “full” status
# information (ExtendedStatus On) or just basic information (ExtendedStatus
# Off) when the “server-status” handler is called. The default is Off.
#
ExtendedStatus On
# Allow server status reports generated by mod_status,
# with the URL of http://servername/server-status
# Uncomment and change the “.example.com” to allow
# access from other hosts.
#
<Location /server-status>
SetHandler server-status
Order deny,allow
Deny from all
Allow from localhost ip6-localhost
<IfModule mod_rewrite.c>
RewriteEngine off
</IfModule>
Allow from 127.0.0.1
# On CentOS / RedHat systems, uncomment the following line
AuthUserFile /etc/httpd/status-htpasswd
# On Debian / Ubuntu systems, uncomment the following line
# AuthUserFile /etc/apache2/status-htpasswd
AuthName “Password protected”
AuthType Basic
Require valid-user
# Allow password-less access for allowed IPs
Satisfy any
</Location>

</IfModule>

Once the configuration is in place, secure the server-status page with a username and password by:

CentOS 6 / CentOS 7 / CentOS 8
[root@web01 ~]# htpasswd -c /etc/httpd/status-htpasswd serverinfo
[root@web01 ~]# service httpd restart

Ubuntu 14.04
[root@web01 ~]# htpasswd -c /etc/apache2/status-htpasswd serverinfo
[root@web01 ~]# a2enconf status.conf
[root@web01 ~]# service apache2 restart

Ubuntu 16.04 and 18.04
[root@web01 ~]# htpasswd -c /etc/apache2/status-htpasswd serverinfo
[root@web01 ~]# /usr/sbin/a2enmod status
[root@web01 ~]# service apache2 restart

You can now visit the following page in your browser to see the output of mod_status:

http://xxx.xxx.xxx.xxx/server-status

Lets say you look at top, and you consistently see an Apache process maxing out a CPU, or using up a ton of memory. You can cross-reference the PID of that Apache child process against the same PID that you find within the server-status page. The requests are constantly changing, so you may need to refresh the /server-status page a couple of times to catch it.

To aid in the troubleshooting as you are trying to match up pids against what is shown in top, you can have the /server-status page refresh automatically by using the following in the URL:

http://xxx.xxx.xxx.xxx/server-status?refresh=2

Once you do locate it, it may give you some idea of what client, or what types of requests, are causing the resource contention issues. Usually it is a specific web application misbehaving, or a specific client is attacking a site.

Identify which node served the content

Troubleshooting a load balanced website can be difficult when trying to narrow down which server is causing the problem. However Apache has the ability to return its hostname via a x-served-by-header, which may help identify the problem node.

To get started, first ensure that Apache will export the hostname within its environment:
## RHEL / CentOS
[root@web01 ~]# vim /etc/sysconfig/httpd
…
export HOSTNAME=$(hostname)
…
## Ubuntu / Debian
[root@web01 ~]# vim /etc/apache2/envvars
…
export HOSTNAME=$(hostname)
…

Now update the site vhost file with the following:

RHEL / CentOS
[root@web01 ~]# vim /etc/httpd/vhost.d/example.com.conf

Ubuntu / Debian
[root@web01 ~]# vim /etc/apache2/vhost.d/example.com.conf

…
<IfModule mod_headers.c>
PassEnv HOSTNAME
Header set X-Served-By “%{HOSTNAME}e”
</IfModule>

Then restart Apache:

RHEL / CentOS
[root@web01 ~]# service httpd restart

Ubuntu / Debian
[root@web01 ~]# service apache2 restart

The hostname should now be returned in the headers are shown below:

[root@web01 ~]# curl -I localhost
HTTP/1.1 200 OK
Date: Sat, 14 Apr 2012 05:27:03 GMT
Server: Apache
Last-Modified: Sat, 14 Apr 2012 05:25:39 GMT
ETag: “5a2ae-0-4bd9cd1c8fac0”
Accept-Ranges: bytes
X-Served-By: web01
Content-Type: text/html; charset=UTF-8

Securing directories with htaccess

There are times when certain directories within a website need to be password protected, or otherwise restricted to specific IP’s. This is useful for a variety of reasons, such as:

• Admin portals such as WordPress’s wp-admin
• Locking down sensitive information such as Munin stats
• Adding an additional layer of security for PHPMyAdmin
• Password protecting an entire site.

Password protect a directory

Password protecting directories within a website is pretty simple. First create a password file for the site by running:

RHEL / CentOS
[root@web01 ~]# htpasswd -c /etc/httpd/example.com-htpasswd username

Ubuntu / Debian
[root@web01 ~]# htpasswd -c /etc/apache2/example.com-htpasswd username

Now append or create an .htaccess file within the desired directory to be password protected by:
[root@web01 ~]# vim /var/www/vhosts/example.com/securedirectory/.htaccess
…
AuthType Basic
AuthName ” Restricted”
AuthUserFile /etc/httpd/example.com-htpasswd
# AuthUserFile /etc/apache2/example.com-htpasswd
Require valid-user
…

Ensure the directory is now password protected by visiting it in your browser and confirming it prompts for credentials and that they work. If it is not working, you will need to ensure that AllowOveride is set to ‘All’ within the Apache vhost configuration as shown below:

<Directory /var/www/vhosts/example.com>
Options -Indexes +FollowSymLinks -MultiViews
AllowOverride All

Restricting directory access by IP address

To simply restrict access to a directory by IP address, append or create an .htaccess file within the desired directory to be restricted by:

[root@web01 ~]# vim /var/www/vhosts/example.com/securedirectory/.htaccess
…
order deny,allow
deny from all
allow from 123.123.123.123

Ensure that directory is now restricted to only allow certain IP’s by visiting the URL in the browser to ensure it returns a forbidden message. Then try adding your IP address temporarily to ensure it now allows you in.

Mod_rewrite

The mod_rewrite modules enables the ability to rewrite incoming requests to different destinations. Rewrite rules can be setup in 2 places:

• The sites vhost configuration file
• Directly within the site’s .htaccess file

Both have some advantages and disadvantages in regards to performance vs each of use that is outlined below:

• Setting up the rules in the .htaccess file allows for changes to be made on the fly without having to restart Apache. This is very useful for customers that are constantly changing their rules. However there is a performance penalty as each time someone visits the site, the . htaccess file must first be processed. So for large rulesets on a high traffic website, this would incur substantial CPU load on the server.
• To avoid the performance penalty, place the rules within the sites vhost configuration file and restart Apache. This causes the rules to be loaded into and served from memory, which negates any CPU penalty.

There is no right way to do this. It really boils down to preference and circumstance.
With that being said, below are some rewrite examples:

Rewrite example.com to www.example.com

RewriteEngine On
RewriteCond %{HTTP_HOST} !^www [NC]
RewriteRule ^(.*)$ http://www.%{HTTP_HOST}$1 [R=301,L]

Force SSL on domain

RewriteEngine On
RewriteCond %{SERVER_PORT} !^443$
RewriteRule ^(.*)$ https://%{HTTP_HOST}$1 [R=301,L]

If using an SSL terminated load balancer, then the rule would need to be:

RewriteEngine On
RewriteCond %{HTTP:X-Forwarded-Proto} !https
RewriteRule ^(.*)$ https://www.domain.com$1 [R=301,L]

Rewrite one domain to another

RewriteEngine on
RewriteCond %{HTTP_HOST} ^(www.)?example.com [NC]
RewriteRule ^(.*)$ http://www.google.com$1 [R=301,L]

Redirect a path to a new domain

Redirect 301 /path http://www.example.com
Redirect 301 /otherpath/somepage.php http://other.example.com

Rewrite page with query string, and strip query string

RewriteEngine On
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteCond %{REQUEST_FILENAME} !-l
RewriteCond %{REQUEST_URI} ^/pages/pages\.php$
RewriteCond %{QUERY_STRING} ^page=[0-9]*$
RewriteRule ^.*$ http://www.example.com/path/? [R=301,L]

Force all URL’s to be lowercase

Please note, this must be in the Apache VirtualHost config, it will not work in the .htaccess.

RewriteEngine On
RewriteMap lc int:tolower
RewriteCond %{REQUEST_URI} [A-Z]
RewriteRule (.*) ${lc:$1} [R=301,L]

Disable TRACE and TRACK methods

RewriteEngine On
RewriteCond %{REQUEST_METHOD} ^(TRACE|TRACK)
RewriteRule .* – [F]

Rewrite images to Cloud Files

Please note, this must be in the Apache VirtualHost config, it will not work in the .htaccess.

<Directory /var/www/vhosts/example.com/content/images>
RewriteEngine On
RewriteRule ^(.*)$ http://c0000.cdn00.cloudfiles.rackspacecloud.com/$1 [R=302,L]
</Directory>

Rewrite all pages to a maintenance page

Options +FollowSymlinks
RewriteEngine On
RewriteBase /
RewriteCond %{REQUEST_URI} !/maintenance.html$
RewriteRule .* /maintenance.html [L]

Rewrite WordPress wp-admin URL requests to master server

WARNING: This one seems to work in test environments, but every WP site has different plugins that do random things. Be sure to have customer test this before blindly putting it in place!
This seems to eliminate the need to use Varnish for admin URL redirection as it satisfies the following requirements:

• Confirm POST requests to wp-admin go to master web server
• Confirm POST requests to admin-ajax.php go to master web server as this is required for WP plugin installation
• Confirm GET requests to admin-ajax.php stay on slave web server to avoid overloading master web server
• Confirm images/media uploads in /wp-admin route to the master web server
• Confirm sessions don’t get lost. Log into wp-admin normally, then switch my /etc/hosts file to point to another slave web server
• Confirm ProxyPreserveHost will actually work and forward over to correct domain on web01Some assumptions that are being made here:
• You have a load balanced WordPress setup with one master web server and multiple slave web servers.
• You have Lsyncd running on the master web server syncing any changes made to the /var/www/vhosts down to the slave web servers

Please note, this must be in the Apache VirtualHost config, not the .htaccess for visibility purposes. This should also
servers only!
In both the 80 and 443 blocks of the Apache VirtualHost config for the website in question, add: only

RewriteEngine On
ProxyPreserveHost On
RewriteCond %{REQUEST_METHOD} =POST
RewriteCond %{REQUEST_URI} /wp-admin/admin-ajax\.php [OR]
RewriteCond %{REQUEST_URI} ^/wp-(admin|login|cron) [OR]
RewriteCond %{HTTP:Content-Type} ^multipart/form-data [NC]
RewriteRule “^/(.*)$” “http://10.x.x.x/$1” [P]

Be sure to update the 10.x.x.x with the internal or service net IP of the master web server, and confirm the slave can access the master over 80 and 443 accordingly in the firewall.

Once this is in place, the customer MUST test every aspect of their site to confirm this works. Things to test include, but are not limited to:

• Confirm images/media can be uploaded via /wp-admin and show up on ALL the web servers after Lsync or Rsync runs
• Confirm sessions in /wp-admin don’t get lost. The primary symptom of this is the user will suddenly be logged out and have to log back in
• Confirm new plugins can be installed and activated via /wp-admin
• Confirm the master server isn’t getting overloaded with requests and the slave web servers are sitting almost idle.

This should work for 90% of the use cases, but there are always those special setups where the rules may need to be tweaked further.

Debugging rewrite rules

It is inevitable that a customer will call with a rewrite rule that is doing something it shouldn’t be doing. Fortunately, Apache has built in debugging for this that simply needs to be enabled. Some important notes about enabling mod_rewrite debugging:

• It must be setup within the Apache vhost for that domain.
• The logging output will be stored in a log file of your choice on the server.
• DO NOT leave the debug log enabled . You will quickly run out of disk space depending on how busy the website is. Disable it before logging off the server!

When you enable the debug log for mod_rewrite, it must be enabled within the Apache vhost configuration file for that domain. From there, the output will be stored in a log file of your choosing.

To enable the mod_rewrite debugging for www.example.com on a CentOS server, do the following:
[root@web01 ~]# vim /etc/httpd/vhost.d/example.com.conf
<VirtualHost *:80>
…
RewriteEngine On
# Uncomment for rewrite debugging
RewriteLog /var/log/httpd/http_rewrite_log
RewriteLogLevel 9
…
<VirtualHost *:443>
RewriteEngine On
# Uncomment for rewrite debugging
RewriteLog /var/log/httpd/http_rewrite_log
RewriteLogLevel 9
…
[root@web01 ~]# apachectl configtest
[root@web01 ~]# service httpd restart
[root@web01 ~]# tail -f /var/log/httpd/http_rewrite_log

To enable mod_rewrite debugging for www.example.com on a Ubuntu server, do the following:
[root@web01 ~]# vim /etc/apache2/sites-enabled/example.com.conf
<VirtualHost *:80>
…
RewriteEngine On
# Uncomment for rewrite debugging
RewriteLog /var/log/apache2/http_rewrite_log
RewriteLogLevel 9
…
<VirtualHost *:443>
RewriteEngine On
# Uncomment for rewrite debugging
RewriteLog /var/log/apache2/http_rewrite_log
RewriteLogLevel 9
…
[root@web01 ~]# apachectl configtest
[root@web01 ~]# service httpd restart
[root@web01 ~]# tail -f /var/log/apache2/http_rewrite_log

Performance

Tuning

Basic memory tuning

Keeping Apache from operating within confines of the available memory is a good starting point for tuning Apache, which is largely controlled by setting MaxClients or MaxRequestsWorkers to a reasonable value.

Despite all the one-liners and tools like ApacheBuddy and Apache2Buddy, none of these tools will be able to tell you exactly what MaxClients needs to be set to. Instead, it simply gives you a starting point. It is up to the system administrator to take into account all the others servers that are running to truly determine how much free memory is available to work with.

### GENERAL FINDINGS & RECOMMENDATIONS ###

Apache2buddy.pl report for server: web01.example.com (123.123.123.123):

Settings considered for this report:
Your server’s physical RAM: 490 MB
Remaining Memory after other services considered: 476 MB
Apache’s MaxClients directive: 16 <——— Current Setting
Apache MPM Model: prefork
Largest Apache process (by memory): 15 MB
[ !! ] Your MaxClients setting is too low.
Your recommended MaxClients setting is between 27 and 30. <——- Acceptable Range (10% of
MAX)
Max potential memory usage: 252 MB
Percentage of TOTAL RAM allocated to Apache: 51.60 %
Percentage of REMAINING RAM allocated to Apache: 53.07 %

A log file entry has been made in: /var/log/apache2buddy.log for future reference.
…

Keep in mind that these are recommendations only! Use common sense before blindly implementing any solutions offered by a script you did not write!

Testing

Benchmarking

Important note:This is ! Rackspace does not perform benchmarking of client applications. This is best left to third party companies UNSUPPORTED that specialize in benchmarking. This information is being displayed here for educational purposes only.

Benchmarking a site is far more than simply throwing requests at the front page of a site. A true benchmark identifying how many visitors a solution
can handle should be custom tailored to:

• Walk a website as the visitors normally would
• Be able to place orders (if applicable)
• Setup new accounts (if applicable)
• Perform administrative functions if the site allows for that
• etc

In short, a true benchmark should be using simulated user activity taken directly from the sites access logs so the test can be as close to real world as possible. Using the various free benchmarking sites on the internet simply test the base page and really does not offer much value since normal day to day traffic rarely just hits a single page.

With that being said, if you wanted to get an idea of how many requests a specific page can handle, you can use Apache Benchmark to get a very rough idea assuming absolutely perfect conditions.

Lets say you wanted to test 5 users each performing 100 requests on the site, you would run:

[root@benchmark01 ~]# ab -n 100 -c 5 http://www.example.com/
…
Concurrency Level: 5
Time taken for tests: 5.024 seconds
Complete requests: 100
Failed requests: 0
Write errors: 0
Total transferred: 13857808 bytes
HTML transferred: 13805288 bytes
Requests per second: 19.90 [#/sec] (mean)
Time per request: 251.211 [ms] (mean)
Time per request: 50.242 [ms] (mean, across all concurrent requests)
Transfer rate: 2693.56 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 58 68 5.0 67 93
Processing: 157 175 12.9 172 219
Waiting: 32 43 10.9 39 84
Total: 222 244 13.4 241 281
Percentage of the requests served within a certain time (ms)
50%     241
66%     248
75%      252
80%     254
90%     265
95%      271
98%      281
99%      281
100%    281 (longest request)

At a very basic level, the results tell me the main page of www.example.com can handle 19.90 requests per second when I have 5 users actively hitting the site. For more detailed information on Apache Benchmark to see what it can really do, check the man pages by:

[root@benchmark01 ~]# man ab

Load testing

First, install siege on a separate server not being tested by:
## CentOS / RedHat 6 & 7
[root@loadtest01 ~]# yum install epel-release
[root@loadtest01 ~]# yum install siege
Ubuntu / Debian
[root@loadtest01 ~]# apt-get update
[root@loadtest01 ~]# apt-get install siege

Now click around the site and grab 10 or more URL’s and post them into a file on your load testing server at

/root/site.txt as shown below:
[root@loadtest01 ~]# vim /root/site.txt
http://example.com
http://example.com
http://example.com/cameras
http://example.com/electronics
http://example.com/home-decor/decorative-accents
http://example.com/home-decor/decorative-accents/herald-glass-vase
http://example.com/apparel/shirts
http://example.com/home-decor/books-music
http://example.com/home-decor/books-music/a-tale-of-two-cities.html
http://example.com/sale.html
http://example.com
http://example.com

Now run a load test. This example will send 50 concurrent connections, with a random delay between 1 and 3 seconds, lasting for 30 minutes against the URL’s posted in /root/site.txt:

[root@loadtest01 ~]# siege -c50 -d3 -t30M -i -f /root/site.txt

While the load test is occurring, things to observe on the server include:

• Check CPU, Memory, IO and overall load on the server
• Check the database to see if there are any intensive queries running frequently, perhaps indicating the need for redisor memcache
• Check the MySQL slow query log to see if there are any queries that need a table index, or otherwise need to be optimized to be less intensive
• Ensure any caching software installed has a good hit rate
• Ensure the site remains online through the load test

Security

Website security is an ever evolving target. Everyday the threat landscape changes, and the security footprint of the application needs to evolve to stay ahead of the threats. Below are some basics for establish a baseline security stance.

Basic Apache Hardening

Below are several of the more common best practices that should be used when attempting to harden Apache. These are by no means the only thing that should be done, but instead is simply serves as a good starting point. First, attempt to mitigate information disc osure that is sought for during recon attempts. This includes:

• Disabling the Trace method
• Limit what can be seen via ServerTokens
• Disable the ServerSignature

These may already be in place in various parts of the Apache configuration, but if they are not set, then they can be setup by:

RHEL / CentOS 6, 7, 8
[root@web01 ~]# vim /etc/httpd/conf.d/security.conf

Ubuntu 14.04, 16.04, 18.04, 20.04 and Debian
[root@web01 ~]# vim /etc/apache2/conf-available/security.conf
# Disable access to the entire file system except for the directories that
# are explicitly allowed later.

# This currently breaks the configurations that come with some web application
# Debian packages. It will be made the default for the release after lenny.

#<Directory />
# AllowOverride None
# Order Deny,Allow
# Deny from all
#</Directory>
# Changing the following options will not really affect the security of the
# server, but might make attacks slightly more difficult in some cases.

# ServerTokens
# This directive configures what you return as the Server HTTP response
# Header. The default is ‘Full’ which sends information about the OS-Type
# and compiled in modules.
# Set to one of: Full | OS | Minimal | Minor | Major | Prod
# where Full conveys the most information, and Prod the least.

ServerTokens Prod

# Optionally add a line containing the server version and virtual host
# name to server-generated pages (internal error documents, FTP directory
# listings, mod_status and mod_info output etc., but not CGI generated
# documents or custom error documents).
# Set to “EMail” to also include a mailto: link to the ServerAdmin.
# Set to one of: On | Off | EMail
#ServerSignature Off

# Allow TRACE method

# Set to “extended” to also reflect the request body (only for testing and
# diagnostic purposes).

# Set to one of: On | Off | extended

TraceEnable Off

Then enable the configuration by:

RHEL / CentOS 6, 7, 8
[root@web01 ~]# service httpd restart

Ubuntu 14.04, 16.04, 18.04, 20.04 and Debian
[root@web01 ~]# a2enconf security
[root@web01 ~]# service apache2 restart

Using secure SSL ciphers and protocols

WARNING: The SSL ciphers and protocols standard changes pretty often, so this section may be quickly outdated. For the most current list of recommended ciphers and protocols, please refer to: Information: Ciphers and PCI Compliance

There is not a one size fits all approach to this. If you have a WAF or IDS in front of your solution, you may be required to have certain ciphers listed so the WAF or IDS can continue to decrypt the traffic for inspection. The same goes for customers that have clients running older browsers and operating systems as they may not support anything about TLSv1.0. Therefore I’ll list out a few common scenerios.

For reference, we generally set the SSL ciphers and protocols in the following location:

• RHEL / CentOS : /etc/httpd/conf.d/ssl.conf
• Ubuntu / Debian : /etc/apache2/mods-enabled/ssl.conf

To secure SSL against many of the currently vulnerabilities out there at the time of this writing, disable TLSv1.0 and enable forward security, you can use:

• SSLCipherSuite EECDH+AESGCM:EECDH+AES256:EECDH+AES128:EDH+AES:RSA+AESGCM:RSA+AES:!ECDSA:!NULL:!MD5:!DSS:!3DES
• SSLProtocol -ALL +TLSv1.1 +TLSv1.2
• SSLHonorCipherOrder On

If there are concerns about disabling TLSv1.0 as its not a hard requirement by the PCI council until 6/2018, and want to keep allowing customers running ancient browsers and operating systems like Windows XP that don’t support anything above TLSv1.0, then you can use:

• SSLCipherSuite ALL:!EXP:!NULL:!ADH:!LOW:!SSLv3:!SSLv2:!MD5:!RC4:!DSS:!3DES
• SSLProtocol all -SSLv2 -SSLv3
• SSLHonorCipherOrder On

If you have an Imperva WAF or Alertlogic IDS in front your solution that needs to decrypt the SSL traffic for analysis, so you therefore can’t use forward security since they need to perform a man-in-the-middle on the traffic, but still want to disable insecure ciphers, then modify the variables in the ssl.conf as follows:

• SSLCipherSuite HIGH:!MEDIUM:!AESGCM:!ECDH:!aNULL:!ADH:!DH:!EDH:!CAMELLIA:!GCM:!KRB5:!IDEA:!EXP:!eNULL:!LOW:!
  RC4:!3DES
• SSLProtocol all -SSLv2 -SSLv3
• SSLHonorCipherOrder On

PCI scan failures

Due to how modern Linux systems backport security updates, PCI scans will frequently show false positives for Apache and PHP.

You can try to determine if your system is truly vulnerable (assuming the system is fully up to date on security patches) by searching for the CVE at the distro’s website:

Redhat / CentOS
https://access.redhat.com/security/cve/<CVE NUMBER>
Ex. https://access.redhat.com/security/cve/CVE-2016-5195

Ubuntu
http://people.canonical.com/~ubuntu-security/cve/

Debian
https://security-tracker.debian.org/tracker/<CVE NUMBER>
Ex. https://security-tracker.debian.org/tracker/CVE-2016-5195

Alternatively, you can try to look through the packages change logs to see if a CVE has already been backported:

RHEL / CentOS
[root@web01 ~]# rpm -q –changelog httpd | grep CVE-2016-5387- add security fix for CVE-2016-5387

Ubuntu / Debian
[root@web01 ~]# zcat /usr/share/doc/apache2/changelog.Debian.gz |grep CVE-2016-8743- debian/patches/CVE-2016-8743.patch: enfore stricter parsing in- CVE-2016-8743

Proactive security

Customers often ask us about security best practices when securing their sites When discussing best security practices for site code, it is important that the customer understands these are tasks their development team needs to make. There is not much that can be done from our end safely as we could break their site accidentally.

Website patch management

If the customer is using an opensource CMS such as WordPress, Drupal, Magento or Joomla, a good starting point for improving a sites securtiy stance is to ensure the CMS is not behind on updates. In many cases, outdated CMS software is a primary means of breaking into a server as it provides the easiest attack vectors for malicious entities. It is very easy for malicious entities to search for published vulnerabilities in the CMS, then simply update their attack signatures and scan the internet looking for sites that match the signature.

Updating the CMS is a customer related task, and therefore should only be performed by the customers developers. The reasons for this include, but are not limited to:

• Plugins/modules may be in use that do not support the updated versions of the CMS.
• Aspects of the website may break for one reason or another, and therefore the updates should be tested in a dev or staging environment by the developer.
• The update could expose a bug within a plugin that creates performance issues that the developer will need to debug.
• We can never be sure what a client customized in the CMS or how they use it, so Rackspace would have no way to know what update might break the site, or what should even be tested to ensure things work.

Reviewing security best practice documentation

Most CMS’s should have some sort of security best practice documentation available. Some quick ones I located are:

• WordPress – http://codex.wordpress.org/Hardening_WordPress
  and http://premium.wpmudev.org/blog/keeping-wordpress-secure-the
  ultimate-guide/?utm_expid=3606929-49.jXdAmx05R2i4dJEDhtW_tA.0
• Drupal – https://www.drupal.org/security/secure-configuration
• Magento – http://docs.magento.com/m1/ce/user_guide/magento/magento-security-best-practices.html
• Joomla – https://docs.joomla.org/Security_Checklist

If the customer wrote their own internet facing application, then they should review:
https://www.owasp.org/index.php/OWASP_Secure_Coding_Practices_-_Quick_Reference_Guide

Web application firewall

To add another layer of security, the customer can explore using a third party service that can provide web application firewall capabilities (WAF). WAF’s act as a firewall of sorts for your websites, blocking common SQL injections, XSS vulnerabilities, etc that may not be patched for whatever reason in the websites code. Such providers of this are:

• https://www.cloudflare.com
• https://www.sucuri.net
• https://www.incapsula.com

For customers with dedicated footprints or that utilize RackConnect, Rackspace also sells dedicated WAF appliances. Check with the customer’s business development consultant for more information on this product offering.

Even with a WAF employed, that is something that should only be used as another part of a defense in depth strategy, when all the sites code base has been updated to the latest possible security release. Also auditing older themes and plugins that have been abandoned by their author is also critical cause that means they are not getting any security updates.

Reactive security

Oftentimes we receive support requests that a website was compromised, and the customer need to know where to begin. Before proceeding, you should always review our official policy that governs compromised servers located here:
https://one.rackspace.com/display/enterprisesupport/Compromised+server+process

But the question typically is, what can be done? Restoring the server or site from backups will simply get the clients back to a state when they were not compromised. But that does not solve the root cause of how they got compromised in the first place, so they will likely just be compromised again very shortly.

Malware scanning

To help point the customer in the right direction, a malware scanner called maldet can be used. As scanning for malware on a customers site is completely unsupported, be sure they sign off on the legal prefab before proceeding.

Instructions for how to install and run a maldet scan can be found here:
Linux sysadmin tools for analyzing compromised servers#Maldet

Search for outdated versions of the CMS

As noted earlier, CMS’s are often compromised as they are running a vulnerable version of the software that already has security updates available, but not applied. As WordPress is an exteremly common CMS, here is how you can quickly determine what versions they are running:

[root@web01 ~]# yum install mlocate
[root@web01 ~]# updatedb
[root@web01 ~]# locate wp-includes/version.php | while read x; do echo -n “$x : WordPress Version ” && egrep
‘^\s*\$wp_version\s*=’ “$x” | cut -d\’ -f2; done | column -t -s :
/var/www/vhosts/example.com/wp-includes/version.php WordPress Version 4.7.2
/var/www/vhosts/example2.com/wp-includes/version.php WordPress Version 4.7.2
/var/www/vhosts/example3.com/wp-includes/version.php WordPress Version 4.3.1
/var/www/vhosts/example4.com/wp-includes/version.php WordPress Version 4.7.2

From there, compare that against the official WordPress documentation to see if it is outdated or not:
https://codex.wordpress.org/WordPress_Versions

The same concept applies for Drupal, Magento and Jooma. Simply locate the version of the software they are running, then present the findings to the customer so their developers can review and take appropriate action.

Review proactive security steps

Finally, run through the previous section on proactive security for a list of additional recommendations located here:
Apache#Proactivesecurity

Troubleshooting

Common Issues

 

Gathering useful information from the logs

The Apache access logs can provide a system administrator with a ton of information about what happened, or what is currently happening. This information could include but is not limited to:

• Determining if a site is a high traffic or low traffic website
• See if one page is receiving a ton of traffic that may need caching
• Checking to see if one IP address is slamming the site with errand traffic
• Determining if there are brute force attacks or something else malicious happening
• Can show how someone compromised a website

Below are some tricks and tips for gathering some useful information from the logs.

Determining which site is the busiest

There are many ways to go about this. However a quick and dirty way to get a quick idea which site is taking on the most traffic is to simple check the size of the access logs by:

[root@web01 ~]# du -sh * |grep access |grep -vE ‘gz|access-|log-‘
297M linux.example.com-access.log
60K fakedomain.example.com-access.log
5.1G example.com-access.log
4.0K console.example.com-access.log
4.0K wiki.example.com-access.log
1.5M store.example.com-access.log

So based off that output, we can reasonably assume that example.com is likely the site taking on the most traffic today assuming log rotation is working of course.

Gathering quick stats about a site

When troubleshooting alarms that occurred earlier in the day, it can be useful to determine inbound connection rates. It may show periods of increased traffic during a specific hour that need to be checked into further.

To get an hourly breakdown of the traffic stats on a specific day, run the following:
[root@web01 ~]# grep “04/May” /var/log/httpd/www.example.com-access.log | cut -d[ -f2 | cut -d] -f1 | awk
F: ‘{print $2″:00″}’ | sort -n | uniq -c
…
32450 11:00
37508 12:00
33701 13:00
45935 14:00
38165 15:00
31950 16:00
38881 17:00
36413 18:00
528201 19:00
42681 20:00
41864 21:00
…

Based off that output, there was a massive spike in connections during the 9:00PM hour (19:00). So break down the 9:00PM hour to show connections per minute:

[root@web01 ~]# grep “04/May/2017:19″ /var/log/httpd/www.example.com-access.log | cut -d[ -f2 | cut -d] -f1
| awk -F: ‘{print $2”:”$3}’ | sort -nk1 -nk2 | uniq -c | awk ‘{ if ($1 > 10) print $0}’
3450 19:00
2774 19:01
3510 19:02
140568 19:03
139728 19:04
142608 19:05
1377 19:06
2333 19:07
1980 19:08
2056 19:09
2123 19:10
…
1997 19:57
1631 19:58
1988 19:59

As shown above, there was some sort of traffic spike that occurred between 9:03PM – 9:05PM. As the window has been narrowed down to a 3 minute
period, more focused analysis can be performed.
Listing the top 10 IP’s accessing the site around 9:03PM:
[root@web01 ~]# grep “04/May/2017:19:03” /var/log/httpd/www.example.com-access.log | awk ‘{print $1}’ | sort -nr | uniq -c |sort -nr | head
Showing the top 10 most called elements on the site at 9:03PM:
Generating traffic analysis report with GoAccess
Important note: This is
[root@web01 ~]# grep “04/May/2017:19:03” /var/log/httpd/www.example.com-access.log | awk ‘{print $7}’ | sort -nr | uniq -c | sort -nr | head
Of course, always check the logs manually as well once the time has been narrowed down as you may identify other patterns to search for.
UNSUPPORTED
! So there are zero guarantees this will work! It is being documented here as many on the floor are using
this software to provide quick traffic analysis for troubleshooting purposes.
GoAccess is an open source real-time web log analyzer and interactive viewer that runs in a terminal in *nix systems or through your browser. It
provides fast and valuable HTTP statistics for system administrators that require a visual server report on the fly. GoAccess parses the specified web
log file and outputs the data on the terminal or out to a html based report that can be provided to the customer. It can provide quick stats in a
consumable format on items such as:
General Statistics
Unique visitors
Requested files
Requested static files
404 or not files that can’t be found
Referrers URL’s
Referring sites
Geo locatition
HTTP status codes
etc
A live demo of GoAccess can be found here:
https://goaccess.io
Installing GoAccess is pretty straight forward. It is already included in most package managers including apt and yum (via EPEL). To install it, simply
run:
## CentOS 6/7 and RHEL 6/7
[root@web01 ~]# yum install goaccess
## Ubuntu and Debian
[root@web01 ~]# apt-get update
[root@web01 ~]# apt-get install goaccess
To run an analysis and output it to the terminal for the interactive report:
[root@web01 ~]# goaccess -f /var/log/httpd/example.com-access.log
Generate an interactive report, outputted as an .html page (report.html) that can be uploaded somewhere and viewed through a browser:
[root@web01 ~]# goaccess -f /var/log/httpd/example.com-access.log –log-format COMBINED -a -o report.html
There are a bunch of options ways to run this. For more information to see everything GoAccess can do, check out the documentation at:
https://goaccess.io/man
Using Apache mod_status to determine whats happening
Apache mod_status can provide a wealth of information including:
The number of workers serving requests
The number of idle workers
The status of each worker, the number of requests that worker has performed and the total number of bytes served by the worker.
A total number of accesses and byte count served.
The time the server was started/restarted and the time it has been running for.
Averages giving the number of requests per second, the number of bytes served per second and the average number of bytes per request.
The current percentage CPU used by each worker and in total by all workers combined.
The current hosts and requests being processed.
For information on how to set it up and use it, refer to: Apache#ConfiguringApachemod_status
Site performance issues
There are a number of things that can lead to performance issues on a website. Some of the more common issues are:
Inefficient queries within MySQL
Buggy, outdated or otherwise poorly written code, plugins or modules
Server issues such as faulty hardware or noisy neighbors if on cloud
Brute force attacks creating resource contention issues
Unexpected traffic spikes due to a high traffic event
Large .htaccess ruleset on a busy site
There is no way to document all these in this article, however they should be pretty self explanatory to get you started in the right direction. While it is
simple to blame the customer’s code and move on, all that does is create mile long tickets that involve multiple techs since we didn’t perform our due
diligence. So some basics to check and document in the ticket include:
Check the MySQL slow query logs for inefficient queries.
Check the MySQL processlist to see if there are queries stacking up, and provide those to the client so their developers can investigate.
Check the web servers access logs to see if its due to a brute force attack, or maybe due to bot traffic.
Check the servers hardware or resources to confirm there are no issues with the drives, CPU, memory, etc.
Confirm the MySQL and Apache tunings are optimal for their setup.
Check the .htaccess ruleset to see if its huge (over a few hundred lines). If so and its a high traffic website, consider moving the rules to the
Apache vhost so they can be loaded into memory instead. See this URL for more information: http://httpd.apache.org/docs/current/howto
/htaccess.html
Document all your findings cleanly in the ticket for the customers developers to review.
Remember, cache is king on any solution. Confirm there is some sort of PHP caching module like APC or OpCode. If there are a lot of database
requests, consider something like Memache or Redis. If the site is a CMS based solution, considering running a caching plugin for the CMS. If there is
a lot of static content, perhaps the site could make use of Varnish, or maybe a CDN provider such as Cloud Flare or Rackspace CDN.
When all else fails, and the issue is not determined to be from a traffic spike, faulty hardware, poor tuning, bad queries in MySQL, etc, then that is
when the customer will need to start profiling their site to narrow the issue down. Some third party tools that are very popular for this is
New Relic
.
If the customer’s developers are unable to locate the issue, we can always throw more powerful hardware at the problem, however that is something
to avoid whenever possible since that is only masking the issue.
Troubleshooting mod_rewrite rules
Troubleshooting mod_rewrite rules can be a science within itself depending on how complicated the ruleset its. Typically though you need to have the
client let you know:
What domain is the issue happening on
How exactly to replicate the issue
What exactly should be happening
From there, its a matter of going through their rewrite rules in the .htaccess file, or within the domain’s Apache vhost configuration file, and checking
for errors or rules that may be overriding the desired rule.
Sometimes you need to enable the mod_rewrite debug log to be able to identify exactly why a rule is not working, or what may be overriding it.
Instructions to enabling the mod_rewrite debug log can be found here: Apache#Debuggingrewriterules
Segmentation faults
More often the not, segmentation faults see within the Apache error logs are caused by an incompatibility with one of the PHP modules. Its rarely the
customers code causing the issue. The usual suspects I have seen in the past include:
APC
Suhosin
NewRelic (When using php53u-suhosin)
Identifying what PHP module is causing the issue can be very difficult. Ideally, the customer’s developers would use a PHP profiler such as xDebug to
narrow down the problem. However its rare for a customer’s developer to know what xDebug is or how to use it. And to make matters more
complicated, xDebug is something that falls far outside what we can support.
To be able to troubleshoot the segmentation fault, you must first figure out how to replicate the segfault. There is no easy way to do this. Typically it
involves surfing the customers site while tailing the Apache error logs and wait for the segfault to trigger. More often than not though, the customer will
state that something is not working on their site, and when you test it, you will see the segmentation fault log to the error logs.
You can attempt to catch the issue with gdb to try to narrow down the problem. Using gdb can be a bit of a pain though as the <packagename>
debuginfo package needs to be installed. Otherwise you get errors about ‘no debugging symbols found’ as shown below:
[root@web01 ~]# gdb -p 13197
…
Reading symbols from /usr/lib64/php/modules/zip.so…(no debugging symbols found)…done.
Loaded symbols for /usr/lib64/php/modules/zip.so
0x00007fa913cb9d9b in accept4 () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install httpd-2.2.15-59.el6.centos.x86_64
The instructions for loading the debug symbols differ depending on what operating system is being used, or if its managed by a subscription manager
or not. The various methods are outlined below:
On dedicated systems managed by a subscription manager such as RHN, simple enable the debug channel according for your operating system. The
channels should be one of the following:
Red Hat Enterprise Linux Server Debuginfo (v. 6).
Red Hat Enterprise Linux Server Debuginfo (v. 7).
For RHEL/CentOS systems NOT managed by a subscription manager, simply enable the debug repo in /etc/yum.repos.d. The example shown is
specific for CentOS 6, but the same concept applies for CentOS 7. Do not change anything else in this file!
[root@web01 ~]# vim /etc/yum.repos.d/CentOS-Debuginfo.repo
[base-debuginfo]
name=CentOS-6 – Debuginfo
baseurl=http://debuginfo.centos.org/6/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-Debug-6
enabled=1
The debuginfo package to install depends on what version of Apache and PHP you are using. As this test server is simply running the distro’s default
versions, we installed the needed package by:
[root@web01 ~]# yum install gdb httpd-debuginfo php-debuginfo
[root@web01 ~]# service httpd restart
If you are running Apache with PHP 5.6 installed from IUS for example, then the installation command would be:
[root@web01 ~]# yum install gdb httpd-debuginfo php56u-debuginfo
[root@web01 ~]# service httpd restart
If you running Ubuntu / Debian with Apache and the stock version of PHP, the debug symbols are installed by:
[root@web01 ~]# apt-get update
[root@web01 ~]# apt-get install gdb apache2-dbg php-phpdbg
[root@web01 ~]# service apache2 restart
And if you are running Ubuntu / Debian with Apache and PHP 7.0, the debug symbols would be installed by:
[root@web01 ~]# apt-get update
[root@web01 ~]# apt-get install gdb apache2-dbg php7.0-phpdbg
[root@web01 ~]# service apache2 restart
Now that the debugging symbols are loaded, gdb becomes more useful. However now we need to know which Apache child pid to connect gdb to.
There are no easy ways to go about this, typically I’ll get approval from the customer to set Apache MaxClients or MaxRequestWorkers to 1, then I
can attach to the single child process as shown below:
[root@web01 ~]# ps waux |grep -E ‘www-data|httpd’
root 13624 0.0 0.5 291148 10324 ? Ss 13:05 0:00 /usr/sbin/httpd
apache 13626 0.0 0.3 291148 5796 ? S 13:05 0:00 /usr/sbin/httpd
root 13638 0.0 0.0 103312 876 pts/0 S+ 13:24 0:00 grep httpd
So know we know the Apache child pid, lets connect gdb to that Apache child process:
[root@web01 ~]# gdb -p 13626
…
whole bunch of info
…
(gdb)
Now watch the Apache error log and wait for another segmentation fault to occur, when it does, go back to your terminal where gdb is sitting, and run
a backtrace by:
(gdb) bt
#0 0xf76ff430 in __kernel_vsyscall ()
#1 0xf7673c99 in __lll_lock_wait () from /lib/i686/cmov/libpthread.so.0
#2 0xf766f104 in _L_lock_936 () from /lib/i686/cmov/libpthread.so.0
#3 0xf766f02e in pthread_mutex_lock () from
/lib/i686/cmov/libpthread.so.0
#4 0xf75fab36 in pthread_mutex_lock () from /lib/i686/cmov/libc.so.6
#5 0xf645ee5d in apc_pthreadmutex_lock () from
/usr/lib/php5/20060613+lfs/apc.so
#6 0xf6457ce6 in apc_cache_info () from
/usr/lib/php5/20060613+lfs/apc.so
#7 0xf64571bb in zif_apc_cache_info () from
/usr/lib/php5/20060613+lfs/apc.so
#8 0xf6977f91 in execute_internal () from
/usr/lib/apache2/modules/libphp5.so
#9 0xf58b18bb in ?? () from /usr/lib/php5/20060613+lfs/suhosin.so
#10 0xffcb0f5c in ?? ()
#11 0x00000001 in ?? ()
#12 0x0000000f in ?? ()
#13 0xffcb0c48 in ?? ()
#14 0x000000a0 in ?? ()
#15 0x0000000e in ?? ()
#16 0x00000000 in ?? ()
The result seems to indicate that APC is deadlocked, probably due to some incapability with suhosin. So now you can try to update those PHP
modules or disable them, then try to replicate the issue again.
Escalation Path
This is an open source project. Therefore there is no formal escalation path. If questions or problems are encountered, try the following:
Check the vendor documentation
Utilize online resources such as google, stackoverflow, etc
Ask around on your team in person, or over IRC/Slack in your teams room
Reach out to an L2, L3, L4 to see if they can point you in the right direction
Reach out on the #linux-techs IRC room or the linux-techs email list
Always remember that not everyone is going to know everything. When asking for help from colleagues in person or through IRC, its always best to
include the following items (where possible) in the question:
What is the problem or error being encountered
How can the issue be replicated
What has already been tried
What online resources have already been checked
References
https://httpd.apache.org/docs