Live log analytics from Apache - format error

@newpipe wrote:

Hi,

I just set up a custom log format in Apache to directly pipe log output to Piwik.

However, I'm getting an Python error from the tool import_logs.py.

My according Apache config looks as follows:


To apache.conf I added:
LogFormat "%v %h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" piwik_log


To my vhost config I added:
CustomLog "|/var/www/vhosts/stats.michi.su/misc/log-analytics/import_logs.py --url=https://stats.michi.su --idsite=4 --debug --recorders=2 --output=/tmp/piwik.log -" piwik_log
and
CustomLog /tmp/horde-temp.log piwik_log


In my debug log I'm getting the following:

2016-06-07 09:10:05,052: [DEBUG] Accepted hostnames: all
2016-06-07 09:10:05,052: [DEBUG] Piwik URL is: https://stats.michi.su
2016-06-07 09:10:05,052: [DEBUG] No token-auth specified
2016-06-07 09:10:05,052: [DEBUG] No credentials specified, reading them from "/var/www/vhosts/stats.michi.su/config/config.ini.php"
2016-06-07 09:10:05,096: [DEBUG] Authentication token token_auth is: removed-manually
2016-06-07 09:10:05,096: [DEBUG] Resolver: static
2016-06-07 09:10:05,213: [DEBUG] Launched recorder
2016-06-07 09:10:05,214: [DEBUG] Launched recorder
2016-06-07 09:10:29,851: [DEBUG] Detecting the log format
2016-06-07 09:10:29,852: [DEBUG] Check format shoutcast
2016-06-07 09:10:29,852: [DEBUG] Error in format checking: Traceback (most recent call last):
File "/var/www/vhosts/stats.michi.su/misc/log-analytics/import_logs.py", line 1957, in check_format
match = candidate_format.check_format(lineOrFile)
File "/var/www/vhosts/stats.michi.su/misc/log-analytics/import_logs.py", line 241, in check_format
file.seek(0)
IOError: [Errno 29] Illegal seek

2016-06-07 09:10:29,852: [DEBUG] Format shoutcast does not match
2016-06-07 09:10:29,852: [DEBUG] Check format iis
2016-06-07 09:10:30,167: [DEBUG] Error in format checking: Traceback (most recent call last):
File "/var/www/vhosts/stats.michi.su/misc/log-analytics/import_logs.py", line 1957, in check_format
match = candidate_format.check_format(lineOrFile)
File "/var/www/vhosts/stats.michi.su/misc/log-analytics/import_logs.py", line 241, in check_format
file.seek(0)
IOError: [Errno 29] Illegal seek

2016-06-07 09:10:30,167: [DEBUG] Format iis does not match
2016-06-07 09:10:30,168: [DEBUG] Check format common_complete

While piping the log directly to the Piwik Python Log parser, Apache also writes the same log to a file. When feeding this file to the Python script, everything works fine:

$ /var/www/vhosts/stats.neurohr.at/misc/log-analytics/import_logs.py --url=https://stats.michi.su --idsite=4 --debug --recorders=2 /tmp/horde-temp.log
2016-06-07 09:55:35,904: [DEBUG] Accepted hostnames: all
2016-06-07 09:55:35,904: [DEBUG] Piwik URL is: https://stats.michi.su
2016-06-07 09:55:35,904: [DEBUG] No token-auth specified
2016-06-07 09:55:35,904: [DEBUG] No credentials specified, reading them from "/var/www/vhosts/stats.neurohr.at/config/config.ini.php"
2016-06-07 09:55:35,950: [DEBUG] Authentication token token_auth is: manually-removed
2016-06-07 09:55:35,950: [DEBUG] Resolver: static
0 lines parsed, 0 lines recorded, 0 records/sec (avg), 0 records/sec (current)
2016-06-07 09:55:35,996: [DEBUG] Launched recorder
2016-06-07 09:55:35,996: [DEBUG] Launched recorder
Parsing log /tmp/horde-temp.log...
2016-06-07 09:55:35,996: [DEBUG] Detecting the log format
2016-06-07 09:55:35,996: [DEBUG] Check format shoutcast
2016-06-07 09:55:35,996: [DEBUG] Format shoutcast does not match
2016-06-07 09:55:35,996: [DEBUG] Check format iis
2016-06-07 09:55:35,996: [DEBUG] Format iis does not match
2016-06-07 09:55:35,996: [DEBUG] Check format common_complete
2016-06-07 09:55:35,996: [DEBUG] Format common_complete matches
2016-06-07 09:55:35,996: [DEBUG] Format match contains 10 groups
2016-06-07 09:55:35,996: [DEBUG] Check format amazon_cloudfront
2016-06-07 09:55:35,996: [DEBUG] Format amazon_cloudfront does not match
2016-06-07 09:55:35,996: [DEBUG] Check format w3c_extended
2016-06-07 09:55:35,996: [DEBUG] Format w3c_extended does not match
2016-06-07 09:55:35,996: [DEBUG] Check format icecast2
2016-06-07 09:55:35,996: [DEBUG] Format icecast2 does not match
2016-06-07 09:55:35,996: [DEBUG] Check format nginx_json
2016-06-07 09:55:35,996: [DEBUG] Format nginx_json does not match
2016-06-07 09:55:35,996: [DEBUG] Check format s3
2016-06-07 09:55:35,996: [DEBUG] Format s3 does not match
2016-06-07 09:55:35,996: [DEBUG] Check format common
2016-06-07 09:55:35,996: [DEBUG] Format common does not match
2016-06-07 09:55:35,996: [DEBUG] Check format common_vhost
2016-06-07 09:55:35,996: [DEBUG] Format common_vhost matches
2016-06-07 09:55:35,996: [DEBUG] Format match contains 8 groups
2016-06-07 09:55:35,996: [DEBUG] Check format ncsa_extended
2016-06-07 09:55:35,996: [DEBUG] Format ncsa_extended does not match
2016-06-07 09:55:35,996: [DEBUG] Format common_complete is the best match

Logs import summary

4 requests imported successfully
0 requests were downloads
3 requests ignored:
    0 HTTP errors
    2 HTTP redirects
    0 invalid log lines
    0 requests did not match any known site
    0 requests did not match any --hostname
    1 requests done by bots, search engines...
    0 requests to static resources (css, js, images, ico, ttf...)
    0 requests to file downloads did not match any --download-extensions

Website import summary

4 requests imported to 1 sites
    1 sites already existed
    0 sites were created:

0 distinct hostnames did not match any existing site:

Performance summary

Total time: 0 seconds
Requests imported per second: 32.08 requests per second

Processing your log data

In order for your logs to be processed by Piwik, you may need to run the following command:
 ./console core:archive --force-all-websites --force-all-periods=315576000 --force-date-last-n=1000 --url='https://stats.michi.su'

What can i do, in order to get the live feeding working?

Thanks,
newpipe

Posts: 1

Participants: 1

Read full topic