We’ve migrated from GA to Matomo with end of May and are experiencing issues with the archiving of the “yearly” period for our two largest sites.
PHP has been configured to not apply memory limits. After processing daily, weekly and monthly periods, the archiving process starts with the yearly period and quickly - in a matter of seconds - eats up all the RAM of the processing host. In our case that is currently 15GB. Before we increase the RAM of the host, we’d like to understand the root cause (and if e.g. doubling it would actually help with the problem).
That’s the trace of the offending command:
# /usr/bin/php7.3 -q /var/www/matomo/console climulti:request -q --matomo-domain='localhost' --superuser 'module=API&method=CoreAdminHome.archiveReports&idSite=3&period=year&date=2021-01-01&format=json&trigger=archivephp&pid=08a5b67142d4621130bbcb821706bf9b4a8501b9570d12a6040db72a59b2ab35ae1421042bbdd1a4187ea36ac48c6e30874c0&runid=27897'
mmap() failed: [12] Cannot allocate memory
mmap() failed: [12] Cannot allocate memory
{"result":"error","message":"Out of memory (allocated 15124066304) (tried to allocate 9134060 bytes) on \/var\/www\/matomo\/core\/DataAccess\/ArchiveSelector.php(378) #0 \/var\/www\/matomo\/core\/ArchiveProcessor.php(213): Piwik\\ArchiveProcessor->aggregateDataTableRecord(name="Goals_ItemsSku") #1 \/var\/www\/matomo\/core\/ArchiveProcessor\/PluginsArchiver.php(168): Piwik\\Plugins\\Goals\\Archiver->callAggregateMultipleReports() #2 \/var\/www\/matomo\/core\/ArchiveProcessor\/PluginsArchiver.php(168): Piwik\\Plugins\\Goals\\Archiver->callAggregateMultipleReports() ","backtrace":"Out of memory (allocated 15124066304) (tried to allocate 9134060 bytes) on \/var\/www\/matomo\/core\/DataAccess\/ArchiveSelector.php(378)\n#0 \/var\/www\/matomo\/core\/ArchiveProcessor.php(213): Piwik\\ArchiveProcessor->aggregateDataTableRecord(name=\"Goals_ItemsSku\")\n#1 \/var\/www\/matomo\/core\/ArchiveProcessor\/PluginsArchiver.php(168): Piwik\\Plugins\\Goals\\Archiver->callAggregateMultipleReports()\n#2 \/var\/www\/matomo\/core\/ArchiveProcessor\/PluginsArchiver.php(168): Piwik\\Plugins\\Goals\\Archiver->callAggregateMultipleReports()\n\n#0 [internal function]: Piwik\\Plugins\\CorePluginsAdmin\\Controller->safemode(Array)\n#1 \/var\/www\/dbuser\/core\/FrontController.php(615): call_user_func_array(Array, Array)\n#2 \/var\/www\/dbuser\/core\/FrontController.php(167): Piwik\\FrontController->doDispatch('CorePluginsAdmi...', 'safemode', Array)\n#3 \/var\/www\/dbuser\/core\/FrontController.php(98): Piwik\\FrontController->dispatch('CorePluginsAdmi...', 'safemode', Array)\n#4 \/var\/www\/dbuser\/core\/FrontController.php(271): Piwik\\FrontController::generateSafeModeOutputFromError(Array)\n#5 [internal function]: Piwik\\FrontController::triggerSafeModeWhenError()\n#6 {main}"}{"result":"error","message":"Out of memory (allocated 15124066304) (tried to allocate 9134060 bytes) on \/var\/www\/matomo\/core\/DataAccess\/ArchiveSelector.php(378) #0 \/var\/www\/matomo\/core\/ArchiveProcessor.php(213): Piwik\\ArchiveProcessor->aggregateDataTableRecord(name="Goals_ItemsSku") #1 \/var\/www\/matomo\/core\/ArchiveProcessor\/PluginsArchiver.php(168): Piwik\\Plugins\\Goals\\Archiver->callAggregateMultipleReports() #2 \/var\/www\/matomo\/core\/ArchiveProcessor\/PluginsArchiver.php(168): Piwik\\Plugins\\Goals\\Archiver->callAggregateMultipleReports() ","backtrace":"Out of memory (allocated 15124066304) (tried to allocate 9134060 bytes) on \/var\/www\/matomo\/core\/DataAccess\/ArchiveSelector.php(378)\n#0 \/var\/www\/matomo\/core\/ArchiveProcessor.php(213): Piwik\\ArchiveProcessor->aggregateDataTableRecord(name=\"Goals_ItemsSku\")\n#1 \/var\/www\/matomo\/core\/ArchiveProcessor\/PluginsArchiver.php(168): Piwik\\Plugins\\Goals\\Archiver->callAggregateMultipleReports()\n#2 \/var\/www\/matomo\/core\/ArchiveProcessor\/PluginsArchiver.php(168): Piwik\\Plugins\\Goals\\Archiver->callAggregateMultipleReports()\n\n#0 [internal function]: Piwik\\Plugins\\CorePluginsAdmin\\Controller->safemode(Array)\n#1 \/var\/www\/dbuser\/core\/FrontController.php(615): call_user_func_array(Array, Array)\n#2 \/var\/www\/dbuser\/core\/FrontController.php(167): Piwik\\FrontController->doDispatch('CorePluginsAdmi...', 'safemode', Array)\n#3 \/var\/www\/dbuser\/core\/FrontController.php(98): Piwik\\FrontController->dispatch('CorePluginsAdmi...', 'safemode', Array)\n#4 \/var\/www\/dbuser\/core\/FrontController.php(271): Piwik\\FrontController::generateSafeModeOutputFromError(Array)\n#5 [internal function]: Piwik\\FrontController::triggerSafeModeWhenError()\n#6 {main}"}
Does the reference to "Goals_ItemsSku’ hint towards an issue with E-Commerce metrics?
We are new to Matomo and it’s data model: Does this provide a hint regarding volume and memory usage of deserializing the raw data?
MariaDB [matomo]> select sum(octet_length(value)) from archive_blob_2021_05 where idsite=3 and name='Goals_ItemsSku';
+--------------------------+
| sum(octet_length(value)) |
+--------------------------+
| 2715132509 |
+--------------------------+
1 row in set (1 min 24.459 sec)
FTR: The site with ID 3 already has it’s own separate archiving process.
1 post - 1 participant