Chamilo : Optimization Guide

Documentation > Optimization Guide

In seldom cases, you will need to start looking into efficiency issues with Chamilo. This guide is a work in progress intended to help administrators optimize their Chamilo installation.

Contents

  1. Using opcaches
  2. Slow queries
  3. Indexes caching
  4. Sessions directories
  5. Users upload directories
  6. Zlib compressed output
  7. Memory considerations for high numbers of users
  8. Avoiding non-fixed values
  9. Speeding file downloads with mod_xsendfile
  10. IGBinary for faster courses backups and better sessions
  11. Removing files download permissions check
  12. MySQL/MariaDB compression
  13. Increasing PHP limits

1. Using opcaches

Zend OpCode (Zend Optimizer+)

From version 5.5, PHP includes the Zend OpCache Optimizer, which can bring considerable efficiency improvements and is very reliable. Using OpCache should come by default, but if you want to make sure it's running, just check that your opcache.ini config file says 
opcache.enable = 1
Some websites will recommend the addition of additional settings, and this is really up to you. Check the official OpCache config page for more information. To check if OpCache is effectively running, you can check the Chamilo systems status page on the administration page, or you can check it in phpinfo, if you have any script with it. Zend OpCache is an "opcode" cache, meaning it will compile static code to make their processing faster. However, this will not allow you to "store" shared variables in memory between all users. To do that, we suggest you complement Zend OpCache (opcode) with a user-land cache like APCu.

APCu

You can also check whether APCu is working or not from the systems status page. Check the official APCu config page for configuration options. In previous versions, this optimization guide contained information about how to use xCache, APC or Memcache to boost the number of online users. However, starting from version 1.11, code has been added to Chamilo to use APCu by default from the banner.lib.php library, so as long as APCu is installed and running, you'll benefit from this optimization naturally.

Other items

It is also worth noting that the Université de Genève, Switzerland, observed that the calculation of the total size used by course documents is one of the heaviest queries in Chamilo, so you might want to cache the results of this one as well, using the same technique.

Finally, if your portal is highly public *and* you are showing the popular courses on the homepage, you might want to also reduce the amount of queries this generates, using the same technique as above, but for the main/inc/lib/auth.lib.php library, looking for the "Tracking::get_course_connections_count()" call:

        while ($row = Database::fetch_array($result)) {
            $row['registration_code'] = !empty($row['registration_code']);
            $count_users = CourseManager::get_users_count_in_course($row['code']);
            $xc = function_exists('apc_exists');
            if ($xc) {
                $apc = apc_cache_info(null, true);
                $apx_end = $apc['start_time']+$apx['ttl'];
                if (apc_exists('my_campus_course_visits_'.$row['code']) AND (time() < $apc_end) AND apc_fetch('my_campus_course_visits_'.$row['code']) > 0) {
                    $count_connections_last_month = apc_fetch('my_campus_course_visits_'.$row['code']);
                } else {
                    $count_connections_last_month = Tracking::get_course_connections_count($row['code'], 0, api_get_utc_datetime(time() - (30 * 86400)));
                    apc_store('my_campus_course_visits_'.$row['code'], $count_connections_last_month, $apc['ttl']);
                }
            } else {
                $count_connections_last_month = Tracking::get_course_connections_count($row['code'], 0, api_get_utc_datetime(time() - (30 * 86400)));
            }
            ...
        }
Finally, the Free Campus of Chamilo has a very specific case of slow query: the courses catalog! Because there might be more than 32,000 courses in there, getting the number of "Connections last month" can be a disastrous query in terms of performances. This is why you should try to cache the results as well.
Obviously, as we are speaking about showing the number of visits this month, it doesn't really matter if the number doesn't refresh for an hour or so...
Locate the main/inc/lib/course_category.lib.php file, open it and go to the browseCoursesInCategory() function.
Locate the $count_connections_last_month = Tracking::get_course_connections_count(...) call, and wrap in into something like this (you'll have to update this to use APCu): 
    $xc = method_exists('Memcached', 'add');
    if ($xc) {
        // Make sure the server is available
        $xm = new Memcached;
        $xm->addServer('localhost', 11211);
        // The following concatenates the name of the database + the id of the
        // access url to make it a unique variable prefix for the variables to
        // be stored
        $xs = $_configuration['main_database'].'_'.$_configuration['access_url'].'_';
    }
    $result = Database::query($sql);
    $courses = array();
    while ($row = Database::fetch_array($result)) {
        $row['registration_code'] = !empty($row['registration_code']);
        $count_users = CourseManager::get_users_count_in_course($row['code']);
        if ($xc) {
            if ($xm->get($xs.'cccount_'.$row['code'])) {
                $number = $xm->get($xs.'cccount_'.$row['code']);
            } else {
                $count_connections_last_month = Tracking::get_course_connections_count($row['code'], 0, api_get_utc_datetime(time() - (30 * 86400)));
                $xm->set($xs.'cccount_'.$row['code'], $count_connections_last_month, 3600);
            }
        } else {
            $count_connections_last_month = Tracking::get_course_connections_count($row['code'], 0, api_get_utc_datetime(time() - (30 * 86400)));
        }
   ...

2. Slow queries

Enable slow_queries in /etc/mysqld/my.cnf, restart MySQL then follow using sudo tail -f /var/log/mysql/mysql-slow.log

In Chamilo 1.9 in particular, due to the merge of all databases into one, you might experience performance issues.
To solve this performance issue, you can execute the following query manually in your database:
ALTER TABLE user_rel_tag ADD INDEX idx_user_rel_tag_user (user_id);


In Chamilo 1.10.0 (the first version of the serie), many indexes were forgotten, so you can boost your database by adding the following indexes:
alter table extra_field_values add index idx_extra_field_values (field_id, item_id);
alter table usergroup_rel_user add index idx_usergroup_ru (usergroup_id);
alter table usergroup_rel_user add index idx_usergroup_ru_u (user_id);
alter table c_student_publication add index idxstudpub_cid (c_id);
alter table c_student_publication add index idxstudpub_uid (user_id);
alter table c_quiz_question add index idx_cqq_cid (c_id);
alter table c_quiz_rel_question ADD INDEX idx_cqrq_qid (question_id);
alter table c_quiz_rel_question ADD INDEX idx_cqrq_cid (c_id);
alter table c_quiz_answer add index idx_qa_cidqid (c_id, question_id);
In Chamilo 1.10.6, two additional queries were confirmed to still have effect a considerable effect: 
ALTER TABLE c_quiz_question_rel_category ADD INDEX idx_qqrc_qid (question_id);
ALTER TABLE c_lp_item_view ADD INDEX idx_clpiv_c_i_v (c_id, id, view_count);
Note that, because these situations only occur when a portal is under real-world high-load stress, we only get to find out about these possible bottlenecks after we release stable versions of Chamilo. This is why we list those queries here. However, as soon as we confirm them with a few real life scenarios, we add them into the core of Chamilo so you can benefit from them immediately by installing a new version.

In Chamilo 1.11.x you can boost the DB tables related surveys invitations by adding the following indexes:

        CREATE INDEX idx_survey_q_qid ON c_survey_question (question_id);
        CREATE INDEX idx_survey_code ON c_survey (code);
        CREATE INDEX idx_survey_inv_code ON c_survey_invitation (survey_code);
        CREATE INDEX idx_survey_qo_qid ON c_survey_question_option (question_id);
        CREATE INDEX idx_c_survey_answerucsq ON c_survey_answer (user, c_id, survey_id, question_id);
Also by adding a index on access_url_rel_session to improve the course/session list 
        CREATE INDEX idx_accessurs_sid ON access_url_rel_session (session_id);
If you have lots of gradebook stuff, add this 
        ALTER TABLE gradebook_result ADD INDEX idx_gb_uid_eid (user_id, evaluation_id);
        ALTER TABLE gradebook_category ADD INDEX idx_gb_cat_parent (parent_id);
        ALTER TABLE gradebook_evaluation ADD INDEX idx_ge_cat (category_id);
        ALTER TABLE gradebook_link ADD INDEX idx_gl_cat (category_id);
If you have to delete a lot of users, you'll realize deleting users is a slow operation. Using the following queries might help: 
        ALTER TABLE extra_field_values ADD INDEX idx_efv_item (item_id);
        ALTER TABLE c_item_property ADD INDEX idx_cip_lasteditu (lastedit_user_id);

3. Indexes caching

One good reference: MySQL documentation on multiple key caches

4. Sessions directories

On large implementations, the users sessions might be stored in numbers too large (hundreds of thousands) to be efficiently managed by the filesystem is stored in one single folder. In order to avoid that, you can either store your sessions in another key-value storage (memcache, redis, etc) or you can instruct PHP to store your session files in a directory with a certain level of subdirectories (so sessions are spread across multiple directories instead of inside just one.

This is done by adding the following setting to your php.ini or your Apache's Virtual Host

php_admin_value session.save_path 1;/var/www/test.chamilo.org/sessions/

Please note that, by defining a different directory than your system's default, you will need to reconfigure your system's session cleaning procedure, which is usually defined under /etc/cron.d/php, so that it cleans this specific directory as well.

5. Users upload directories

The default in Chamilo is now to spread user accounts in 10 different directories inside app/upload/users/ to avoid overloading that specific directory. Nothing to be done here. Please move on.

6. Zlib compressed output

Although this will not make your server faster, compressing the pages you are sending to the users will definitely make them feel like your website's responses are a lot faster, and thus increase their well-being when using Chamilo.

Zlib output compression has to be set at two levels: PHP configuration for PHP pages and Apache for images and CSS.

To update the PHP configuration (either in php.ini or in your VirtualHost), use the zlib.output_compression. If you set this inside your Apache's VirtualHost, you should use the following syntax. 
php_value zlib.output_compression 1

Configuring your Apache server to use output compression is a bit trickier. You have to use the mod_deflate module to do it. Your configuration should look like something like this (please read the corresponding documentation before implementing in production).
Easy mode: 
AddOutputFilterByType DEFLATE text/html text/plain text/xml
or, for every content type (dangerous) you can put the following inside a location or directory block:
SetOutputFilter DEFLATE

Advanced mode: 

# Insert filter
SetOutputFilter DEFLATE

# Netscape 4.x has some problems...
BrowserMatch ^Mozilla/4 gzip-only-text/html

# Netscape 4.06-4.08 have some more problems
BrowserMatch ^Mozilla/4\.0[678] no-gzip

# MSIE masquerades as Netscape, but it is fine
# BrowserMatch \bMSIE !no-gzip !gzip-only-text/html

# NOTE: Due to a bug in mod_setenvif up to Apache 2.0.48
# the above regex won't work. You can use the following
# workaround to get the desired effect:
BrowserMatch \bMSI[E] !no-gzip !gzip-only-text/html

# Don't compress images
SetEnvIfNoCase Request_URI \
\.(?:gif|jpe?g|png)$ no-gzip dont-vary

# Make sure proxies don't deliver the wrong content
Header append Vary User-Agent env=!dont-vary
Don't have time or resources to optimize your Chamilo installation yourself? Hire an official Chamilo provider and get it sorted out professionally by specialists. Valid XHTML 1.0 Transitional Valid CSS

Memory considerations for high numbers of users

Some administration scripts *have to* handle lists of all users, and this might have a considerable impact on portals with very high numbers of users. For example, the main/admin/add_users_to_session.php script that handles the registration of users into a specific session, if used with the (non-default) full list of users, will devour about 3KB per user, which, for 100,000 users, translates into the need for around 300MB of RAM just to show this page, and to around 3GB for 1,000,000 users.
This mode is not loaded by default, but could still be selected, leading to a "Fatal error: Allowed memory size ... exhausted" message.
The only non-scripted solution here is to allow for the corresponding amount of RAM for your PHP configuration (memory_limit = 300M) or your specific VirtualHost if you use mod-php5 (php_value memory_limit 300M).

Avoiding dynamic values

Many things in Chamilo are written focusing on the ease of use, even for the administrator. Sometimes, these settings are weighing a little bit more on the system. This is the case, between others, of the mail.conf.php file (being loaded unconditionally) and its CONSTANT "IS_WINDOWS_OS", which is defined by a function call (api_is_windows_os()) at the beginning of main_api.lib.php. The definition of this constant (which is executed at *every* page load) can easily be avoided, and the only place where it is used unconditionally (mail.conf.php) can be modified to set the line as you expect it (depending on whether you use sendmail/exim or smtp). 
$platform_email['SMTP_MAILER']       = 'smtp';
or 
$platform_email['SMTP_MAILER']       = 'mail';
In fact, the complete loading of mail.conf.php can also be avoided if loaded conditionally (with require_once) when sending an e-mail (which is the only case where it is useful).

As an additional node, on very active portals with a lot of courses for each users, the icons that appear next to the courses illustrating changes in the corresponding course might be heavyweighted. You can alter slightly the behaviour by not querying for notifications you don't care about, like dropbox, notebook or chat. Change this in main/inc/lib/display.lib.php, in function show_notification().

Speeding file downloads with mod_xsendfile

It might have come to your attention that file downloads through Chamilo might get slow, under default conditions, in particular using Apache 2.

There are several ways to fix this, one of which is removing the .htaccess inside the courses/ directory. This, however, will remove all permissions checks on the files contained in this directory, so... most of the time, not ideal unless your portal is *really* open to the world.

Another technique, revealed to us by VirtualBlackFox on this Stackoverflow post, is to use the X-SendFile module for Apache 2.2+ (other web servers might offer other solutions, or avoid the problem initially).

Installing the X-SendFile module will depend on your operating system, but if you use Ubuntu, you'll have to check you are including the "universe" repository inside your packages sources (check /etc/apt/sources.list), then: 

sudo apt-get update
sudo apt-get install libapache2-mod-xsendfile
sudo service apache2 restart
Once you're done with installing, you'll have to configure Chamilo to use it.
First, edit your VirtualHost or your Apache configuration in general (in Ubuntu, check the /etc/apache2/ or /etc/apache2/sites-available/ folder). This is done by adding the following line inside your configuration, and reloading Apache (example provided on the basis of a virtual host located in /etc/apache2/sites-available/my.chamilo.net.conf) : 
sudo vim /etc/apache2/sites-available/my.chamilo.net.conf
# add the following line:
  X-SendFile on
# exit the file
sudo service apache2 reload
Finally, you'll have to got to your Chamilo configuration file, and add the following line at the very bottom of the file main/inc/conf/configuration.php: 
$_configuration['enable_x_sendfile_headers'] = true;
Done! Now your downloads should go substantially faster. This is still a feature in observation. We're not sure the benefits are sufficient, so don't hesitate to let us know in the related issue in Chamilo's tracking system

IGBinary for courses backups and better sessions management

IGBinary is a small PECL library that replaces the PHP serializer. It uses less space (so less memory for serialized objects) and is particularly efficient with memory-based storages (like Memcached). Use it for course backups (see issue 4443) or to boost sessions management.

Removing files download permissions check

This measure is not cumulative with mod_xsendfile explained above. It is not *recommended* either, as it removes an important security layer.

In Chamilo, for security and tracking purposes, all downloaded files pass through PHP scripts that check whether the user has access to the file given his/her current permissions. This process requires important database accesses and processing, which might terminally affect your server's performance. In particular, this can have a huge effect if having hundreds of simultaneous users accessing learning paths pages composed of local resources.

The logic behind this verification is that, whatever resources that needs to be downloaded/viewed that come from the /courses/ directory, the /courses/.htaccess file with get in the middle and redirect these accesses to a PHP script (usually called download.php but there are more than one depending on the type of resource).

If you want to speed up files accesses and you don't really care about whom can see your files, then an option is to simply change this redirection to download.php and let Apache treat the file directly.

Furthermore, using a PHP script for the download (unless you have special rules) will usually prevent static content caching, which will multiply downloads and use large amounts of additional bandwidth.

Typically, the .htaccess will look like this (with additional comments):


RewriteEngine On
RewriteBase /courses/
RewriteCond %{REQUEST_URI} !^/main/
RewriteRule ([^/]+)/document/(.*)&(.*)$ $1/document/$2///$3 [N]
RewriteRule ([^/]+)/scorm/(.*)$ /main/document/download_scorm.php?doc_url=/$2&cDir=$1 [QSA,L]
RewriteRule ([^/]+)/document/(.*)$ /main/document/download.php?doc_url=/$2&cDir=$1 [QSA,L]
RewriteRule ([^/]+)/work/(.*)$ /main/work/download.php?file=work/$2&cDir=$1 [QSA,L]

The idea is to allow direct access (without access validation) to resources in the "scorm" directory with: 

RewriteEngine On
RewriteBase /courses/
RewriteCond %{REQUEST_URI} !^/main/
RewriteRule ([^/]+)/document/(.*)&(.*)$ $1/document/$2///$3 [N]
RewriteRule ([^/]+)/scorm/(.*)$ /app/courses/$1/scorm/$2 [QSA,L]
RewriteRule ([^/]+)/document/(.*)$ /main/document/download.php?doc_url=/$2&cDir=$1 [QSA,L]
RewriteRule ([^/]+)/work/(.*)$ /main/work/download.php?file=work/$2&cDir=$1 [QSA,L]

This is easy, doesn't require a server reload and you should see the results pretty quickly. As mentioned above, if security of your content is an issue, though, you should avoid using this technique.

You can also mitigate the risk by disabling permissions check only for some static resource like css,js and fonts files.
For that is required to load header module in apache (check with a2enmod in your favorite root terminal)
add theses line after RewriteBase /courses/: 

<IfModule mod_headers.c>
    # all file name ended with these  extensions names will bypass the permission check   (and also served by the browser cache at  the next request)
    <FilesMatch "\.(gif|jpg|jpeg|png|js|pdf|ico|icon|css|swf|avi|mp3|ogg|wav|ttf|otf|eot|woff)$">
        Header unset Cache-Control
        Header set Cache-Control "public, max-age=29030400"
        RequestHeader unset Cookie
        Header unset ETag
    </FilesMatch>
</IfModule>
# also adjust files here
RewriteRule (\.(html|gif|jpg|jpeg|png|js|pdf|ico|icon|css|swf|avi|mp3|ogg|wav|ttf|otf|eot|woff))$ - [L]

Since version 1.11.10, the .htaccess has been modified to do this by default with media files, and a change post-1.11.10 also does it for documents that are not in a SCORM folder. These changes will improve speed considerably but will lower the security on media files, as a direct link could be used to open the file with no validation. As such, you can comment those lines with media files to ensure increased security, at the cost of performance. These are the two lines (followed by their more wide-ranging rule) that have to be present in .htaccess for maximum efficiency. 

RewriteRule ^courses/([^/]+)/scorm/(.*([\.js|\.css|\.png|\.jpg|\.jpeg|\.gif]))$ app/courses/$1/scorm/$2 [QSA,L]
RewriteRule ^courses/([^/]+)/scorm/(.*)$ main/document/download_scorm.php?doc_url=/$2&cDir=$1 [QSA,L]
[...]
RewriteRule ^courses/([^/]+)/document/(.*([\.js|\.css|\.png|\.jpg|\.jpeg|\.gif|\.mp4|\.webm|\.avi|\.mpeg|\.mp3|\.wav|\.ogg]))$ app/courses/$1/document/$2 [QSA,L]
RewriteRule ^courses/([^/]+)/document/(.*)$ main/document/download.php?doc_url=/$2&cDir=$1 [QSA,L]

MySQL/MariaDB compression

If your database server is separate from your web server, you have to play with bandwidth, firewalls, and network restrictions in general.
In particular, when dealing with large-scale portals, the time a SQL query will take to return to the web server will take longer and, eventually, in the most critical cases, will take too long, and your web servers will be completely overloaded (load average very high because the system is waiting for I/O operations, but processors usage not being very high is a clear sign of this).
To solve this kind of issues, MySQL and MariaDB offer a data compression mechanism, which will reduce the amount of data passed between PHP and the database server. Ultimately, this reduction will lower bandwidth usage and reduce the impact of numerous and heavy data requests (and save you).
In 1.10.0, we have added the possibility to enable this compression very easily, from the configuration.php file, uncommenting the following line: 

//$_configuration['db_client_flags'] = MYSQL_CLIENT_COMPRESS;
This should have an immediate effect on the load average on your server.

Increasing PHP limits

As your use of Chamilo increases and you get above the thousands of users, you're likely to hit a few milestones set by PHP to avoid hacks. One of them is PHP5.4's Suhosin extension limit post_max_vars, which was extended into PHP5.5 and above through the max_input_vars limit. This limit is usually set to 1000. What does it mean?
It means that, when you manipulate any list greater than 1000 items, PHP will automatically remove anything sent above the first 1000 registers (usually a little bit less because it needs to add the other input fields of the page). For example, if subscribing 5 new users to a course where you already have 1000 users subscribed, you will remain at 1000, although the 1000 will not necessarily be the 1000 that were there in the first place (they are sent in order of the elements inside the form, so probably alphabetically, depending on the page).

Increasing this limit to a higher level (say 10,000 instead of 1000) should be relatively safe, considering your application is normally not open to the public (and so also open to the evil kind of users). So, in your php.ini, this limit should now look like this:

    max_input_vars = 10000


A number of other limits might also become an issue in the long run, like memory_limit, post_max_size, etc. We have given reasonnable recommendations in the installation process for these values, but remember that if you have a larger portal than anyone else, you probably need to give it more care than anyone else.

Authors

Don't have time or resources to optimize your Chamilo installation yourself? Hire an official Chamilo provider and get it sorted out professionally by specialists.
Valid XHTML 1.0 Transitional Valid CSS