Parsing logs very slow when using multithread import script

@Velborg wrote:

Hello,

i am doing some test because i have high traffic website log to parse and i need it to be fast so i want to parse 2 differents logs at the same time.

my server is a 12vCPU, 16G RAM nginx/php-fpm/mysql and i am usually using the command line without problem :
./import_logs.py --url=piwik_url --idsite=ID --debug --debug --recorders=10 --recorder-max-payload-size=150 --enable-http-errors --enable-http-redirects /my_log

For my tests i used a unique site, changed the recorders but seems like it's very slow parsing 2 logs at the same time :
- parsing a single log to the site i am using the command :
./import_logs.py --url=piwik_url --idsite=ID --debug --debug --recorders=5 --recorder-max-payload-size=150 --enable-http-errors --enable-http-redirects /my_log
the execution time seem "correct" :
Performance summary

---

Total time: 36 seconds
Requests imported per second: 373.17 requests per second

• parsing two logs for the same site at the same time :
  commands used are :
  ./import_logs.py --url=piwik_url --idsite=ID --debug --debug --recorders=5 --recorder-max-payload-size=150 --enable-http-errors --enable-http-redirects /my_log1
  ./import_logs.py --url=piwik_url --idsite=ID --debug --debug --recorders=5 --recorder-max-payload-size=150 --enable-http-errors --enable-http-redirects /my_log2

results for the log1 :
Performance summary

---

Total time: 369 seconds
Requests imported per second: 37.93 requests per second

results for the log2 :
Performance summary

---

Total time: 370 seconds
Requests imported per second: 37.97 requests per second

I even caught an error 500 when parsing the logs at the same time but the auto retry was fine.

So what did i do wrong ? Can we parse multi logs at the same time ? Why are the parsing very very slow (36 sec parsing one log => 370 seconds by pasing 2 logs simultaneously) ?

Posts: 1

Participants: 1

Read full topic