[REQUEST] Common Log Format for AMaViS* ?

[Mark Lawrence]

Dear developers,

As the creator of the amavis-stats package I am making a humble request
for all flavors of amavis to standardize on a single log format for
base-level messages.

My reasons for asking this are hopefully quite obvious: A tool which
attempts to generate statistics from a log file cannot easily do a good
job when the formats are not well defined. It also appears that the
formats have a tendency to change with some releases...

There are potentially other minor side effects such as better understood
user reports, easier trouble-shooting across different versions, or the
ability to compare combinations of amavis's and scanners... etc.

I don't think that the actual coding to conform to a standard would be a
huge amount of work, but not something I am qualified for. What I can
offer however is to drive the process to define a format.

I would suggest that the scope of such a project is limited to the basic
detection messages, which tend to contain the same type of information
across flavors.

For example: A single log message conforming to the standard could be
constructed according to the following definition:

- left bracket
- unique process/message id
- right bracket
- one or more spaces
- one of:
- Passed
- INFECTED (<space separated virus names>)
- SPAM (<test causing failure>)
- BANNED (<name/type>)
- Other...
- source address
- ->
- destination address
- further details...
- etc

Messages that don't conform simply have another format specific to the
flavour or log level, but do not include the tokens specified above. Or,
you could potentially go a bit further and specify tags for log level
LOG1, LOG2 or ERROR, or I don't know what.

What I would like is some feedback on if this idea is supported in
general, and if so what peoples thoughts are on what should be defined. If
some form of standard format is implmemented in all versions, then I can
produce a better amavis-stats which can easily be used by everyone.

Regards,
Mark.
--
Mark Lawrence (***@null.net)



-------------------------------------------------------
This SF.net email is sponsored by: SF.net Giveback Program.
SourceForge.net hosts over 70,000 Open Source Projects.
See the people who have HELPED US provide better services:
Click here: http://sourceforge.net/supporters.php
_______________________________________________
AMaViS-user mailing list
AMaViS-***@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/amavis-user
AMaViS-FAQ:http://www.amavis.org/amavis-faq.php3
AMaViS-HowTos:http://www.amavis.org/howto/

[Mark Martinec]

Mark Lawrence,

| Dear developers,
| As the creator of the amavis-stats package I am making a humble request
| for all flavors of amavis to standardize on a single log format for
| base-level messages.

I can't speak for others, I'll only comment on what applies to amavisd-new.

| It also appears that the formats have a tendency to change
| with some releases...

Guilty, but speaking of the main log entry at log level 0
(the user-configurable $log_templ, see README.customize), only some
minor additions were made. Probably the main problem with this log entry
is that not all information of interest is available through macros,
so other log entries may have to be parsed to obtain such information,
which is clumsy. I have this on a to-do list.

| ... What I can offer however is to drive the process to define a format.

I welcome this effort.

| I would suggest that the scope of such a project is limited to the basic
| detection messages, which tend to contain the same type of information
| across flavors.
| For example: A single log message conforming to the standard could be
| constructed according to the following definition:

It is close to the current amavisd-new log. I'll comment in-place:

| - left bracket
| - unique process/message id
| - right bracket

parenthesis instead of brackets are used at the moment,
and this is not user-configurable.

| - one or more spaces

| - one of:
| - Passed
| - INFECTED (<space separated virus names>)
| - SPAM (<test causing failure>)
| - BANNED (<name/type>)
| - Other...

Currently the following are used (with examples):
Passed,
Not-Delivered,
INFECTED (Exploit.IFrame.Gen, Exploit.IFrame.Gen)
BANNED name/type (application/x-msdownload, metro.exe),

I can change 'BANNED name/type' to 'BANNED' to avoid spaces in the keyword.

The 'Not-Delivered' is currently a descriptor for any reason of
non-delivery apart from virus and banned. I can easily split out
a further reason: BAD-HEADER. I did't dare to use 'SPAM' instead of
'Not-Delivered', as spaminess is an infividual matter for each recipient,
and we are only producing one main log entry per message (not one per
recipient). When more macros will be available, it may be possible
to declare SPAM with more certainty.

The above list can be changed to the following even now (as it is
user-configurable):

Passed,
INFECTED (...),
BANNED (...),
BAD-HEADER,
SPAM,

For each of them, the list of reasons is optional,
and if present, it is surrounded in parenthesis.

At the moment no special effort is made to protect parenthesis or other
characters within the (...) lists. It may be useful to require
balanced nested parentheses if present, or somehow protected characters,
e.g. \( and \), if present within the list.

| - source address
| - ->
| - destination address

recipients are a comma-separated list,
each recipient is in angle brackets,
no spaces outside angle brackets (but possible iinside),

| - further details...
| - etc

Remaining optional information fields are:

quarantine qid,
Message-ID: <mid>,
Resent-Message-ID: <mid>,
Hits: n

For historic reasons 'quarantine' is not followed by a colon.
All other optional fields are (and will be) followed by colon.


| (<space separated virus names>)

The lists are space-comma -separated at the moment, not space -separated.

It is easy to change that:
Replace %V with [<%V>| ]
and similar for others.
Although if this is done, possible spaces within each element
would better be protected/encoded somehow.


Here is a new definition of $log_templ (to be place in amavisd.conf),
- to replace 'BANNED name/type' with 'BANNED',
- to replace 'Not-Delivered' with 'SPAM' (not always true, but hey...)
- to factor-out BAD-HEADER
- and to add a colon after 'quarantine':

$log_templ = '[? %#V |[? %#F |[? %#X |[? %#D |SPAM|Passed]|BAD-HEADER]|BANNED (%F)]|INFECTED (%V)], #
<%o> -> [<%R>|,][? %i ||, quarantine: %i], Message-ID: %m[?%r||, Resent-Message-ID: %r], Hits: %c';



Mark


-------------------------------------------------------
This SF.net email sponsored by: Enterprise Linux Forum Conference & Expo
The Event For Linux Datacenter Solutions & Strategies in The Enterprise
Linux in the Boardroom; in the Front Office; & in the Server Room
http://www.enterpriselinuxforum.com
_______________________________________________
AMaViS-user mailing list
AMaViS-***@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/amavis-user
AMaViS-FAQ:http://www.amavis.org/amavis-faq.php3
AMaViS-HowTos:http://www.amavis.org/howto/

[Moshe Jacobson]

Hi guys,

Post by Mark Martinec
$log_templ = '[? %#V |[? %#F |[? %#X |[? %#D |SPAM|Passed]|BAD-HEADER]|BANNED (%F)]|INFECTED (%V)], #
<%o> -> [<%R>|,][? %i ||, quarantine: %i], Message-ID: %m[?%r||, Resent-Message-ID: %r], Hits: %c';

I'd like to suggest the $log_templ that I use, which provides all
information necessary, in a relatively straightforward and uniform
format:

$log_templ = '[? %#V |[? %#F |[?%#D|SPAM|Passed]|BANNED (%F)]|INFECTED (%V)], #
<%o> -> [<%R>|,], hits=%c, tests=[%M| ][? %i ||, quarantine=%i][? %m ||, msgid=%m]';

The lines look something like this when logged:

SPAM:
amavisd[16986]: (16986-02) SPAM, <***@x.com> -> <***@x.net>, hits=5.402, tests=BAYES_80 BIZ_TLD HTML_40_50 HTML_MESSAGE MIME_HTML_MOSTLY, quarantine=bf788974e9dbef197a3ef2d26de55cfc-20031019-000039-16986-02, msgid=<b39e01c395f5$934adcc4$***@1e8reh1>

VIRUS:
amavisd[16983]: (16983-28) INFECTED (W95/CIH-10xx), <***@x.be> -> <***@x.net>, hits=-, tests=, quarantine=virus-20031019-000530-16983-28, msgid=<***@mirapoint3.x.be>

BANNED:
amavisd[8811]: (08811-17) BANNED ("snow city.jpg.pif"), <***@x.COM> -> <***@x.net>, hits=-, tests=, quarantine=virus-20031019-101808-08811-17

Passed:
amavisd[16984]: (16984-12) Passed, <***@x.x.x.us> -> <***@x.net>, hits=2.369, tests=BAYES_20 HTML_20_30 HTML_FONTCOLOR_UNKNOWN HTML_FONT_BIG HTML_MESSAGE HTML_TAG_BALANCE_BODY HTML_TAG_BALANCE_HTML NO_REAL_NAME, msgid=<***@x.x.x.us>

This provides most, if not all, of the information I need in one line.

Feel free to modify it to suit your own needs, or to come up with a
standard format.

By the way, I did add the %M macro to the definition of %builtins in
the amavsid script, so it shows the SpamAssassin rules matched on that
message, so that they could be shown on that one line. Now I'm trying
to remember the mnemonic that I used (how did I get %M for that??),
but I can't remember it! haha.

Anyway, hope this helps.

Moshe
--
Moshe Jacobson
http://runslinux.net
AIM: Jehsom



-------------------------------------------------------
This SF.net email sponsored by: Enterprise Linux Forum Conference & Expo
The Event For Linux Datacenter Solutions & Strategies in The Enterprise
Linux in the Boardroom; in the Front Office; & in the Server Room
http://www.enterpriselinuxforum.com
_______________________________________________
AMaViS-user mailing list
AMaViS-***@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/amavis-user
AMaViS-FAQ:http://www.amavis.org/amavis-faq.php3
AMaViS-HowTos:http://www.amavis.org/howto/

[Ralf Hildebrandt]

Post by Moshe Jacobson
$log_templ = '[? %#V |[? %#F |[?%#D|SPAM|Passed]|BANNED (%F)]|INFECTED (%V)], #
<%o> -> [<%R>|,], hits=%c, tests=[%M| ][? %i ||, quarantine=%i][? %m ||, msgid=%m]';

This doesn't log the tests for me:

Oct 27 16:49:51 hauptpostamt amavis[15511]: (15511-11) Passed,
<***@netscape.net> -> <***@charite.de>, hits=22.59,
tests=, msgid=<***@mr5.bezeqint.net>
--
Ralf Hildebrandt (Im Auftrag des Referat V a) ***@charite.de
Charite - Universitätsmedizin Berlin Tel. [phone]
Gemeinsame Einrichtung von FU- und HU-Berlin Fax. [phone]
Referat V a - Kommunikationsnetze - AIM. ralfpostfix


-------------------------------------------------------
This SF.net email is sponsored by: The SF.net Donation Program.
Do you like what SourceForge.net is doing for the Open
Source Community? Make a contribution, and help us add new
features and functionality. Click here: http://sourceforge.net/donate/
_______________________________________________
AMaViS-user mailing list
AMaViS-***@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/amavis-user
AMaViS-FAQ:http://www.amavis.org/amavis-faq.php3
AMaViS-HowTos:http://www.amavis.org/howto/

[Moshe Jacobson]

Ralf,

Post by Moshe Jacobson
$log_templ = '[? %#V |[? %#F |[?%#D|SPAM|Passed]|BANNED (%F)]|INFECTED (%V)], #
<%o> -> [<%R>|,], hits=%c, tests=[%M| ][? %i ||, quarantine=%i][? %m ||, msgid=%m]';

Go back and read my original message. I said I had to add a line to
the amavisd program to add support for %M in the $log_templ.

Moshe
--
Moshe Jacobson
http://runslinux.net
AIM: Jehsom



-------------------------------------------------------
This SF.net email is sponsored by: The SF.net Donation Program.
Do you like what SourceForge.net is doing for the Open
Source Community? Make a contribution, and help us add new
features and functionality. Click here: http://sourceforge.net/donate/
_______________________________________________
AMaViS-user mailing list
AMaViS-***@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/amavis-user
AMaViS-FAQ:http://www.amavis.org/amavis-faq.php3
AMaViS-HowTos:http://www.amavis.org/howto/

[Ralf Hildebrandt]

Post by Moshe Jacobson
$log_templ = '[? %#V |[? %#F |[?%#D|SPAM|Passed]|BANNED (%F)]|INFECTED (%V)], #
<%o> -> [<%R>|,], hits=%c, tests=[%M| ][? %i ||, quarantine=%i][? %m ||, msgid=%m]';

a) Where is the logging format documented?
b) Right now I try to log the tests always, no matter what happens
with the message using this:

$log_templ = '[? %#V |[? %#F |[?%#D|Not-Delivered|Passed]|BANNED name/type (%F)]|INFECTED (%V)], #
<%o> -> [<%R>|,][? %i ||, quarantine %i], Message-ID: %m, Hits: %c, Tests: %M';

But %M is always empty, resulting in:

Oct 27 17:02:34 hauptpostamt amavis[19931]: (19931-07) Passed,
<***@chellaston.co.uk> -> <***@charite.de>, Message-ID:
<***@chellaston.co.uk>, Hits:
20.664, Tests:

:(
--
Ralf Hildebrandt (Im Auftrag des Referat V a) ***@charite.de
Charite - Universitätsmedizin Berlin Tel. [phone]
Gemeinsame Einrichtung von FU- und HU-Berlin Fax. [phone]
Referat V a - Kommunikationsnetze - AIM. ralfpostfix


-------------------------------------------------------
This SF.net email is sponsored by: The SF.net Donation Program.
Do you like what SourceForge.net is doing for the Open
Source Community? Make a contribution, and help us add new
features and functionality. Click here: http://sourceforge.net/donate/
_______________________________________________
AMaViS-user mailing list
AMaViS-***@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/amavis-user
AMaViS-FAQ:http://www.amavis.org/amavis-faq.php3
AMaViS-HowTos:http://www.amavis.org/howto/

[Mark Martinec]

Moshe,

| I'd like to suggest the $log_templ that I use, which provides all
| information necessary, in a relatively straightforward and uniform
| format:
| $log_templ = '[? %#V |[? %#F |[?%#D|SPAM|Passed]|BANNED (%F)]|INFECTED (%V)], #
| <%o> -> [<%R>|,], hits=%c, tests=[%M| ][? %i ||, quarantine=%i][? %m ||, msgid=%m]';

I don't mind the '=' instead of ': ', but it is perhaps better
to avoid unnecessary changes for precisely the reasons you described.
Still, users are able to change the template to their liking
or to your instructions in log-analyzing software.

Now that we tried to change 'Not-delivered' to 'SPAM',
it becomes even more obvious how wrong it is. It is trying
to stash two independent pieces of information into a single label.
E.g. a spam, tagged but passed, is seen in this log entry as 'Passed',
not as spam.

Similarly, BANNED or VIRUS or BAD-HEADER does not tell if the
mail was delivered (e.g. to a virus lover), or not.

So this is my next approximation:

$log_templ = '[?%#D|Blocked|[?%#N|Passed|Partial]] #
[? %#V |[? %#F |[? %#X |[? %k |CLEAN|SPAM]|BAD-HEADER]|BANNED (%F)]|INFECTED (%$
, <%o> -> [<%R>|,]#
[? %i ||, quarantine: %i]#
, Message-ID: %m#
[? %r ||, Resent-Message-ID: %r]#
, Hits: %c';


It distinguishes for example:
Blocked SPAM (to a normal user)
Passed SPAM (to a spam lover)
Partial SPAM (some recipients wanted it, some not)
Passed CLEAN (normal)
Passed BAD-HEADER (e.g. bad headers, but $final_bad_header_destiny=D_PASS)


Btw, it relies on a new macro %k, which indicates the number of
recipients for which spam level was above kill level.
This is from my current code, the lookup call needs to be adjusted
for the current released version:

'k'=> sub { scalar(grep # any recipient declared the message be killed ?
{ !$_->recip_whitelisted_sender &&
($_->recip_blacklisted_sender ||
defined $spam_level &&
$spam_level>=lookup($_->recip_addr,@spam_kill_level_maps))
} @{$MSGINFO->per_recip_data}) },
'1'=> sub { scalar(grep # above tag level for any recipient?
{ !$_->recip_whitelisted_sender &&
($_->recip_blacklisted_sender ||
defined $spam_level &&
$spam_level>=lookup($_->recip_addr,@spam_tag_level_maps))
} @{$MSGINFO->per_recip_data}) },
'2'=> sub { scalar(grep # above tag2 level for any recipient?
{ !$_->recip_whitelisted_sender &&
($_->recip_blacklisted_sender ||
defined $spam_level &&
$spam_level>=lookup($_->recip_addr,@spam_tag2_level_maps))
} @{$MSGINFO->per_recip_data}) },



Mark


-------------------------------------------------------
This SF.net email sponsored by: Enterprise Linux Forum Conference & Expo
The Event For Linux Datacenter Solutions & Strategies in The Enterprise
Linux in the Boardroom; in the Front Office; & in the Server Room
http://www.enterpriselinuxforum.com
_______________________________________________
AMaViS-user mailing list
AMaViS-***@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/amavis-user
AMaViS-FAQ:http://www.amavis.org/amavis-faq.php3
AMaViS-HowTos:http://www.amavis.org/howto/

[Paul Gamble - MWeb]

Hi

Whilst amavis-new logging format is being discussed, may I raise a concern/feature-request in this regard? I am only concerned with the Spam-filtering functionality of amavis-new.

My problem is that amavis-new does not log on a per-recipient basis and I am hoping that consideration could be given to providing this capability. My experience with amavis-new (v20030616-p5 btw) shows me that for a message with multiple recipients where some of these recipients are opted in to Spam checking while others are opted out (or have the sender white-listed, etc), amavis-new only logs a single line of either "Passed" for ALL recipients OR "Not-delivered" for ALL recipients. I haven't yet figured out when or why amavis-new sometimes elects to log this with a "Passed" line and sometimes with a "Not-delivered" line. The point is that it only includes one or the other, but not both, and that the line included lists all the recipient addresses.

Similarly confusing/misleading, is the "SPAM" log line that lists ALL the recipients instead of just those recipients that want Spam filtering.

Essentially, the problem I have is the requirement of being able to provide a per-recipient count of Spam messages filtered and need a way to derive this from the amavis-new log.

If anyone can comment on this or point the way to a solution it would be greatly appreciated. I am trying very hard to avoid customizing the code...

Many thanks,
Paul Gamble

-----Original Message-----
From: Mark Martinec [mailto:***@ijs.si]
Sent: Saturday, October 18, 2003 2:42 AM
To: amavis-***@lists.sourceforge.net
Subject: Re: [AMaViS-user] [REQUEST] Common Log Format for AMaViS* ?


Mark Lawrence,

| Dear developers,
| As the creator of the amavis-stats package I am making a humble request
| for all flavors of amavis to standardize on a single log format for
| base-level messages.

I can't speak for others, I'll only comment on what applies to amavisd-new.

 
M-Web: S.A.'s most trusted and reliable Internet Service Provider. Just Like That.

To join, go to: http://join.mweb.co.za or call 0860032000.


-------------------------------------------------------
This SF.net email is sponsored by OSDN developer relations
Here's your chance to show off your extensive product knowledge
We want to know what you know. Tell us and you have a chance to win $100
http://www.zoomerang.com/survey.zgi?HRPT1X3RYQNC5V4MLNSV3E54
_______________________________________________
AMaViS-user mailing list
AMaViS-***@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/amavis-user
AMaViS-FAQ:http://www.amavis.org/amavis-faq.php3
AMaViS-HowTos:http://www.amavis.org/howto/

[Mark Martinec]

Paul,

| My problem is that amavis-new does not log on a per-recipient basis and
| I am hoping that consideration could be given to providing this
| capability. My experience with amavis-new (v20030616-p5 btw) shows me
| that for a message with multiple recipients where some of these
| recipients are opted in to Spam checking while others are opted out (or
| have the sender white-listed, etc), amavis-new only logs a single line
| of either "Passed" for ALL recipients OR "Not-delivered" for ALL
| recipients. I haven't yet figured out when or why amavis-new sometimes
| elects to log this with a "Passed" line and sometimes with a
| "Not-delivered" line. The point is that it only includes one or the
| other, but not both, and that the line included lists all the recipient
| addresses.

This will change with the next major release. If recipients form
two groups (one with mail being delivered, the other not), there will be
two top-level log entries, each one pertaining to its subgroup, e.g.:

(19028-03) Passed SPAM, <sender> -> <recip-lover1>,
quarantine:..., Message-ID: <...>, Hits: 21.781

(19028-03) Blocked SPAM, <sender> -> <recip2>,<recip3>,
quarantine:..., Message-ID: <...>, Hits: 21.781

If one or the other group will be empty, the corresponding log entry
will be left out, so usually there will only be one or the other entry.

| Similarly confusing/misleading, is the "SPAM" log line that lists ALL
| the recipients instead of just those recipients that want Spam
| filtering.

This is just an informative log entry at log level 1.
At the time in the code when it is produced, it is not yet
fully known what will happen later at the delivery time.

The log processing should mostly depend on the main log entry
at log level 0, formed from the $log_templ template, late in the
mail processing, when most information is already available.
The plan is to make some more information available through macros.

As a preview/illustration, this is the value of $log_templ that I'm using
right now to see is I like it:

[?%#D||Passed #
[? %#V |[? %#F |[? %#X |[? %k |CLEAN|SPAM]|BAD-HEADER]|BANNED (%F)]|INFECTED (%$
, <%o> -> [%D|,]#
[? %q ||, quarantine: %i]#
[? %m ||, Message-ID: %m]#
[? %r ||, Resent-Message-ID: %r]#
, Hits: %c#
]
[?%#O||Blocked #
[? %#V |[? %#F |[? %#X |[? %k |CLEAN|SPAM]|BAD-HEADER]|BANNED (%F)]|INFECTED (%$
, <%o> -> [%O|,]#
[? %q ||, quarantine: %i]#
[? %m ||, Message-ID: %m]#
[? %r ||, Resent-Message-ID: %r]#
, Hits: %c#
]


It depends on two minor modifications to the program (allowing multiline
text to produce multiple log entries, and on a new %O macro, returning
a list of 'other-than delivered' recipients).

Mark


-------------------------------------------------------
This SF.net email is sponsored by: The SF.net Donation Program.
Do you like what SourceForge.net is doing for the Open
Source Community? Make a contribution, and help us add new
features and functionality. Click here: http://sourceforge.net/donate/
_______________________________________________
AMaViS-user mailing list
AMaViS-***@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/amavis-user
AMaViS-FAQ:http://www.amavis.org/amavis-faq.php3
AMaViS-HowTos:http://www.amavis.org/howto/