CPU load problem analysis

To perform the analysis we will install the sysstat passthrough if this is not already present.

$ apt-cache search iostat sysstat - system performance tools for Linux $ sudo apt-get install sysstat

To continue we will put a php page not performing, and I did it: D … I wanted a page that takes a lot of CPU. So I create a page that displays 100 times a random number of 10000 characters. We will go back for the beauty of the operation, but the result is the rendezvous the CPU increases: D.

$ sudo vim /data/vhosts/siteA/docroot/perf/random.php

If you go on the URL: https://www.linux202-sitea.com/perf/random.php, you will have a list of character, if you click to refresh the page other number will be present.

We will now support the server, we will visualize the behavior with the vmstat command:

[client]$ ab -n 200 -c 10 -l https://www.linux202-siteA.com/perf/random.php Concurrency Level: 10 Time taken for tests: 20.719 seconds Complete requests: 200 Failed requests: 0 Total transferred: 100047400 bytes HTML transferred: 100000000 bytes Requests per second: 9.65 [#/sec] (mean) Time per request: 1035.974 [ms] (mean) Time per request: 103.597 [ms] (mean, across all concurrent requests) Transfer rate: 4715.49 [Kbytes/sec] received Connection Times (ms) min mean[+/-sd] median max Connect: 64 143 36.9 139 293 Processing: 400 883 129.8 897 1265 Waiting: 141 545 96.4 551 889 Total: 540 1026 141.6 1038 1430 Percentage of the requests served within a certain time (ms) 50% 1038 66% 1088 75% 1116 80% 1141 90% 1183 95% 1224 98% 1334 99% 1398 100% 1430 (longest request) $ vmstat 5 20 procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu----- r b swpd free buff cache si so bi bo in cs us sy id wa st 13 0 940 951656 74600 503100 0 0 266 37 322 525 26 3 70 1 0 11 0 940 947320 74600 503140 0 0 0 0 1228 3242 94 6 0 0 0 10 0 940 946772 74624 503164 0 0 0 13 1271 3252 94 6 0 0 0 11 0 940 955588 74648 503192 0 0 0 12 1172 3224 94 6 0 0 0 5 0 940 955292 74672 503208 0 0 0 11 1286 3170 94 6 0 0 0 0 0 940 959788 74696 503236 0 0 0 14 812 1145 8 3 89 0 0 1 0 940 961992 74720 503248 0 0 0 13 727 899 3 1 96 0 0

Of course you can also use the command top to visualize the behavior, I wanted to introduce you another possible method that is well script and provides all the information of the system (Memory, Input / Ouput, cpu …). If you want to use top in script it is also possible with the -b option for batch which is very practical.

# Avant test de charge $ uptime 17:11:59 up 22 min, 1 user, load average: 0.96, 1.62, 1.42 # Après test de charge $ uptime 17:13:02 up 23 min, 1 user, load average: 2.81, 1.96, 1.55

In the following situation, what is the solution?

• Correct the php code, it would be the solution to put in place because clearly there is a problem, however this is not always possible at our level

So what is the workaround that can be put in place?

• Of course this depends on your application, in this case the question that I can ask myself: is the random number must change each refresh? Can I set up a cache system so that it is generated only at a defined interval?

We could modify the php code so that it adds a header (header) html so that the client hides the content of the page. This is a very good solution unfortunately as the cache system will be done at the client if I have a lot of query from multiple source, caching operation of the page will NOT be done. At least not as we hear, the load on the server will be the same, it will solve the problem if the same client / browser accesses the page.

So we will introduce the cache system at the server level, the result will be:

• The server will generate the result of the php dynamic page once

• Subsequently the apache server will hide the content in this memory

• In future requests, the same content will be sent to the applicant

• Once the cache expires the page will be regenerated and hidden, so on.

Of course the negative side of this operation is that the dynamic content will not be generated at each request, but if we think a little if we take a news site such as lemonde.fr, lapresse.ca, … If we use a 5-minute cache system to answer thousands of requests, is it really critical?!?!

Using mod_cache

The module that we will use for our needs will be mod_cache, we will see only a flyover, however this will offer you the possibility to go further by yourself.

The system mod_cache function with 2 mode:

• mod_cache_disk implements a disk storage manager. The headers and bodies are stored separately on the disk in a directory structure based on the md5 digest of the cached URL. Multiple Negotiated Content Responses can be stored at the same time, but partial content caching is not supported by this module. The htcacheclean utility allows you to list and delete cached URLs, and to keep the cache below certain size and number of inode limits.

• mod_cache_socache Implements a storage manager based on a shared object cache. Headers and bodies are stored together under a single key based on the URL of the cached response. Negotiated multiple-content responses can be stored simultaneously, but this module does not support caching of partial content.

We will start with the first is very equivalent with the second so no stress: D. Warning the name of the module for hidden in memory is not the same with the version apache 2.2.

We will activate the modules that is already available under Ubuntu.

$ ls /etc/apache2/mods-available/*cache* /etc/apache2/mods-available/authn_socache.load /etc/apache2/mods-available/file_cache.load /etc/apache2/mods-available/cache.load /etc/apache2/mods-available/socache_dbm.load /etc/apache2/mods-available/cache_disk.conf /etc/apache2/mods-available/socache_memcache.load /etc/apache2/mods-available/cache_disk.load /etc/apache2/mods-available/socache_shmcb.load /etc/apache2/mods-available/cache_socache.load $ sudo a2enmod cache Enabling module cache. To activate the new configuration, you need to run: service apache2 restart $ sudo a2enmod cache_disk Considering dependency cache for cache_disk: Module cache already enabled Enabling module cache_disk. To activate the new configuration, you need to run: service apache2 restart $ ls /etc/apache2/mods-enabled/*cache* cache.load cache_disk.conf cache_disk.load socache_shmcb.load

Let’s look at the cache configuration file for the hard drive that is set by default without the comments:

$ cat /etc/apache2/mods-enabled/cache_disk.conf | grep -v "#" | grep -v "^$" CacheRoot /var/cache/apache2/mod_cache_disk CacheDirLevels 2 CacheDirLength 1

We have the configuration of the directory that will be used to store cached content this via the CacheRoot directive. The CacheDirLevels directive sets the number of subdirectory levels that the cache will contain. The CacheDirLength directive sets the number of characters that each subdirectory name in the cache hierarchy will contain.

We have the configuration of the directory that will be used to store cached content this via the CacheRoot directive. The CacheDirLevels directive sets the number of subdirectory levels that the cache will contain. The CacheDirLength directive sets the number of characters that each subdirectory name in the cache hierarchy will contain.

If we look at the directory defined by CacheRoot we will find that the permissions are set for the user running the web server:

$ ls -ld /var/cache/apache2/mod_cache_disk drwxr-xr-x 2 www-data www-data 4096 Jan 14 12:46 /var/cache/apache2/mod_cache_disk

If you choose to change this configuration do not forget to adjust the configurations accordingly.

At this point even if I redo a load test I would have exactly the same behavior as before because although the modules are loaded and configure “globally” no virtual server has to activate it! So we will enable it for the siteA virtual server.

$ sudo vim /etc/apache2/sites-enabled/siteA-ssl.conf [... OUTPUT COUPE ...] Alias "/cm-images" "/data/vhosts/common/images" # Mise en cache de tous les contenus CacheEnable disk / Options none AllowOverride ALL Require all granted [... OUTPUT COUPE ...]

We validate the configuration and restart, I have to restart, because there are modules to load.

$ sudo apachectl configtest && sudo /etc/init.d/apache2 restart Syntax OK * Restarting web server apache2 ...done.

We activated the cache and performed the restart of apache, go on the pages of the site.

• https://www.linux202-sitea.com/perf/random.php

• https://www.linux202-sitea.com/

With the htcacheclean command you can see the content that is cached:

$ sudo htcacheclean -A -p /var/cache/apache2/mod_cache_disk/ https://www.linux202-sitea.com:443/Free_Software_Foundation_logo.png? 620 37866 200 0 1466682926812864 1466769326812864 1466682926812463 1466682926812864 1 0 https://www.linux202-sitea.com:443/index.html? 612 109 200 0 1466682926780187 1466769326780187 1466682926779031 1466682926780187 1 0 $ sudo htcacheclean -a -p /var/cache/apache2/mod_cache_disk/ https://www.linux202-sitea.com:443/Free_Software_Foundation_logo.png? https://www.linux202-sitea.com:443/index.html?

WARNING: Do not forget to use the command with sudo otherwise your user does not have the permission to access the content you will have a result EMPTY.

why do not we see the PHP page in the system the list of hidden pages?!?! We will have to put a small modification in the php page, we will have to define HTML header. Here is what we will add:

$ sudo vim /data/vhosts/siteA/docroot/perf/random.php <?php function generateRandomString($length = 100000) { [... OUTPUT COUPÉ ...]

We have 2 header: TODO: to complete

• Cache-Control: must-revalidate, max-age = 3600 *:

• Vary: Accept-Encoding:

We validate: D, we return to the page and we reuse the command:

$ sudo htcacheclean -a -p /var/cache/apache2/mod_cache_disk/ https://www.linux202-sitea.com:443/perf/random.php [... OUTPUT COUPÉ ...]

It’s time to repeat a performance test: D.

[client]$ ab -n 200 -c 10 -l https://www.linux202-siteA.com/perf/random.php Concurrency Level: 10 Time taken for tests: 4.369 seconds Complete requests: 200 Failed requests: 0 Total transferred: 2134200 bytes HTML transferred: 2080000 bytes Requests per second: 45.78 [#/sec] (mean) Time per request: 218.458 [ms] (mean) Time per request: 21.846 [ms] (mean, across all concurrent requests) Transfer rate: 477.02 [Kbytes/sec] received Connection Times (ms) min mean[+/-sd] median max Connect: 34 190 86.2 209 328 Processing: 1 20 18.1 14 93 Waiting: 1 13 14.2 9 79 Total: 35 210 94.7 239 341 Percentage of the requests served within a certain time (ms) 50% 239 66% 283 75% 297 80% 302 90% 313 95% 320 98% 325 99% 330 100% 341 (longest request) $ uptime # avant 17:25:22 up 35 min, 1 user, load average: 0.70, 0.85, 1.06 $ vmstat 5 20 procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu----- r b swpd free buff cache si so bi bo in cs us sy id wa st 1 0 940 877884 75524 520964 0 0 174 28 365 604 25 3 71 1 0 10 0 940 875124 75524 520992 0 0 0 1 844 1999 64 4 31 0 0 0 0 940 876540 75548 521004 0 0 0 30 855 1650 33 4 64 0 0 # Apres $ uptime 17:26:16 up 36 min, 1 user, load average: 1.06, 0.96, 1.09

We can say that it WALK: D, woot !! Well it’s not a reason to program like a pig ;-).

I told you this is just an introduction to the system we will all take the time to explore a little options available one is in must enjoy: D.

When we activate a cache system the first question we often ask is how long will the system hide the information?!?! This is defined by the CacheDefaultExpire directive