Discussion:
What am I going to break: CC_CLEAN,7
(too old to reply)
Michael Scheidell
2009-01-23 22:05:52 UTC
Permalink
I am implementing, or could work with you Mark, if enough people need
it, a seperate set of catagries for 'content' filtering.
Tried to add a catagory CC_CONTENT { 10 } but amavisd logs the 'tagged'
emails as 'Passed INFECTED'.

Trying (now) using CC_CLEAN,7 and CC_SPAMMY,7 and CC_SPAM,7.
Doesn't look like (yet) that I am stepping on anything.

Am I? what will 2.6.3 break? are you working on something anyway?

(yes, ugly, why bother?). Two reasons:

Schools seem to want to know SPAM vs CONTENT, want the option of
'tagging' (reporting), quarantining and/or blocking emails in or out
that are in their list of dirty words.

SA is easy, just take their list and make a xxx_content.cf out of it.

amavisd-new would happily do its thing, marking, taging, quarantining,
dropping the email based on its total score.

What I am looking to do is be able to tag/quarantine, mark DIFFERENTLY,
based on if the email hit:
SPAM+CONTENT, spam only, CONTENT ONLY, etc.

might try different msgs.content and quar_type values later.

to that end, I am experimenting with amavisd.custom. (I selected BAYES
just for a test, custom xxx_content.cf rules would be easy for me to
identify)

have this in amavisd.custom:

sub CC_CLEAN() { 1 };
sub CC_SPAMMY() { 5 };
sub CC_SPAM () { 6 };
sub new {
my($class,$conn,$msginfo) = @_;
my($self) = bless {}, $class;
$self; # returning an object activates further callbacks,
# returning undef disables them
}

sub checks { # may be left out if not needed
my($self,$conn,$msginfo) = @_;
my($ll) = 3;
my($status) = $msginfo->spam_status;
do_log($ll,"CUSTOM: checks: status: $status");
if ($status =~ /BAYES_...[0-9]/) {
do_log($ll,"CUSTOM: checks: BAYES");
$msginfo->add_contents_category(CC_CLEAN,7);
}
}
sub before_send {
my($self,$conn,$msginfo) = @_;
my($ll) = 3; # log level (0 is the most important level, 1, 2,... 5
less so)
do_log($ll,"CUSTOM: before_send");
my($bayes) = $msginfo->is_in_contents_category(CC_CLEAN,7);
do_log(3, "CUSTOM: bayes = $bayes");
/* maybe do different things to msgs.quar_type, msgs.content.
}
--
Michael Scheidell, CTO
Phone: 561-999-5000, x 1259
*| *SECNAP Network Security Corporation
* Certified SNORT Integrator
* King of Spam Filters, SC Magazine 2008
* Information Security Award 2008, Info Security Products Guide
* CRN Magazine Top 40 Emerging Security Vendors
* Finalist 2009 Network Products Guide Hot Companies

_________________________________________________________________________
This email has been scanned and certified safe by SpammerTrap(r).
For Information please see http://www.secnap.com/products/spammertrap/
_________________________________________________________________________

------------------------------------------------------------------------------
This SF.net email is sponsored by:
SourcForge Community
SourceForge wants to tell your story.
http://p.sf.net/sfu/sf-spreadtheword
_______________________________________________
AMaViS-user mailing list
AMaViS-***@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/amavis-user
AMaViS-FAQ:http://www.amavis.org/amavis-faq.php3
AMaViS-HowTos:http://www.amavis.org/howto/
Mark Martinec
2009-01-27 14:10:17 UTC
Permalink
Michael,
Post by Michael Scheidell
I am implementing, or could work with you Mark, if enough people need
it, a seperate set of catagries for 'content' filtering.
Tried to add a catagory CC_CONTENT { 10 } but amavisd logs the 'tagged'
emails as 'Passed INFECTED'.
The CC_* contents should be renumbered some day with some free range
between the values to allow inserting new contents categories.
So far there hasn't been any pressings needs, the minor
contents categories usually suffice.
Post by Michael Scheidell
Trying (now) using CC_CLEAN,7 and CC_SPAMMY,7 and CC_SPAM,7.
Doesn't look like (yet) that I am stepping on anything.
That's fine.

With the exception of CC_BADH, currently the only used minor ccats
are 0 and sometimes a 1. I sometimes use 2 and 3 with CC_CLEAN
in my custom hooks (by-sender and by-recipient quarantines).

The CC_BADH uses 0..8 for the moment:
ccat_min 0: other, 1: bad MIME, 2: 8-bit char, 3: NUL/CR,
4: empty, 5: long, 6: syntax, 7: missing, 8: multiple
Post by Michael Scheidell
Am I? what will 2.6.3 break?
No.
Post by Michael Scheidell
are you working on something anyway?
No. I was thinking of how to represent finer classifications
of contents as provided by a CRM114, but haven't done any
work in that direction. For example, bounces might be a
useful separate category, independent from spam/clean/virus.
Just a thought...
Post by Michael Scheidell
Schools seem to want to know SPAM vs CONTENT,
What the hack is a CONTENT? Nonempty mail?
Post by Michael Scheidell
want the option of
'tagging' (reporting), quarantining and/or blocking emails in or out
that are in their list of dirty words.
Ah, a CONTENT is polite word for dirty words :)
Post by Michael Scheidell
SA is easy, just take their list and make a xxx_content.cf out of it.
amavisd-new would happily do its thing, marking, taging, quarantining,
dropping the email based on its total score.
What I am looking to do is be able to tag/quarantine, mark DIFFERENTLY,
SPAM+CONTENT, spam only, CONTENT ONLY, etc.
could
might try different msgs.content and quar_type values later.
to that end, I am experimenting with amavisd.custom. (I selected BAYES
just for a test, custom xxx_content.cf rules would be easy for me to
identify)
sub CC_CLEAN() { 1 };
sub CC_SPAMMY() { 5 };
sub CC_SPAM () { 6 };
Just import declarations near the beginning of custom hooks:

BEGIN {
import Amavis::Conf qw(:platform :confvars c cr ca);
import Amavis::Util qw(do_log untaint);
};
Post by Michael Scheidell
sub new {
my($self) = bless {}, $class;
$self; # returning an object activates further callbacks,
# returning undef disables them
}
sub checks { # may be left out if not needed
my($ll) = 3;
my($status) = $msginfo->spam_status;
do_log($ll,"CUSTOM: checks: status: $status");
if ($status =~ /BAYES_...[0-9]/) {
do_log($ll,"CUSTOM: checks: BAYES");
$msginfo->add_contents_category(CC_CLEAN,7);
}
}
Normally the same contents category should be set both as
a global per-message attribute, as well as per-recipient
attributes. Sometimes one can get away with just one or the other,
but setting them both is the safest way, preventing any surprises:

$msginfo->add_contents_category(CC_CLEAN,7);
for my $r (@{$msginfo->per_recip_data}) {
$r->add_contents_category(CC_CLEAN,7);
}

Mark


------------------------------------------------------------------------------
This SF.net email is sponsored by:
SourcForge Community
SourceForge wants to tell your story.
http://p.sf.net/sfu/sf-spreadtheword
_______________________________________________
AMaViS-user mailing list
AMaViS-***@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/amavis-user
AMaViS-FAQ:http://www.amavis.org/amavis-faq.php3
AMaViS-HowTos:http://www.amavis.org/howto/
Michael Scheidell
2009-01-27 15:48:43 UTC
Permalink
Post by Mark Martinec
Post by Michael Scheidell
Schools seem to want to know SPAM vs CONTENT,
What the hack is a CONTENT? Nonempty mail?
Ah, a CONTENT is polite word for dirty words :)
And, as an extention, 'DLP' (data leak prevention)
Clients want to block/tag/mark/quarantine email that might contain specific
(regexp matched) information.

For example:

A bank (in the us) wants to flag as '[PRIVATE]' any INCOMING email that
might contain a social security number, or bank account number.
(they really don't want to bounce it back as 'SPAM'..) GLBA rules seem to
hint that the bank is legally responsible to encrypt that email and even
hide it from 'normal' employies.

A publically traded company want to flag as '[PRIVATE]' any (in or out)
email that might deal with a possible merger during a quiet period.

Yes, messy, but they don't want it handled like a 'virus' or 'spam'.
Post by Mark Martinec
Normally the same contents category should be set both as
a global per-message attribute, as well as per-recipient
attributes. Sometimes one can get away with just one or the other,
Thanks for the help. I am including my investigations into this on the
public list so that others who might, in the future come up with
requirements like this may find a starting point.
--
Michael Scheidell, CTO
Post by Mark Martinec
|SECNAP Network Security
Finalist 2009 Network Products Guide Hot Companies
FreeBSD SpamAssassin Ports maintainer


_________________________________________________________________________
This email has been scanned and certified safe by SpammerTrap(r).
For Information please see http://www.secnap.com/products/spammertrap/
_________________________________________________________________________

------------------------------------------------------------------------------
This SF.net email is sponsored by:
SourcForge Community
SourceForge wants to tell your story.
http://p.sf.net/sfu/sf-spreadtheword
_______________________________________________
AMaViS-user mailing list
AMaViS-***@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/amavis-user
AMaViS-FAQ:http://www.amavis.org/amavis-faq.php3
AMaViS-HowTos:http://www.amavis.org/howto/
Mark Martinec
2009-01-27 16:33:37 UTC
Permalink
Michael,
Post by Mark Martinec
Normally the same contents category should be set both as
a global per-message attribute, as well as per-recipient
attributes. Sometimes one can get away with just one or the other,
but setting them both is the safest way, preventing any surprises.
As an example where the above may not be true is a per-recipient
bypass_*_checks. While a message may be globally flagged as spam,
some recipients may decide to turn a blind eye on spam results
(match @bypass_spam_checks_maps), so a per-message contents category
($msginfo->...) could be a CC_SPAM (if at least one recipient wanted
spam checking), but for users bypassing spam checks their
per-recipient ($r->...) setting would NOT be flagged as CC_SPAM, and
their passed message would therefore not have a header tagged as spam.

Per-recipient contents category is observed for per-recipient
actions, like adding/editing header fields, or quarantining.

A global setting would be observed for actions on a message
as a whole, like logging, where an attempt is made to somehow
summarize actions for each recipient of a message.

Mark

------------------------------------------------------------------------------
This SF.net email is sponsored by:
SourcForge Community
SourceForge wants to tell your story.
http://p.sf.net/sfu/sf-spreadtheword
_______________________________________________
AMaViS-user mailing list
AMaViS-***@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/amavis-user
AMaViS-FAQ:http://www.amavis.org/amavis-faq.php3
AMaViS-HowTos:http://www.amavis.org/howto/
Michael Scheidell
2009-02-02 21:10:50 UTC
Permalink
99% where I want to be.

I want to set msgs.content = 'T' if CC_CLEAN,7. do I do it in
custom-after_send() or custom_end()?

is the (persistant?) db connection still available after_send() or end()?


this works:

package Amavis::Custom;
use strict;
use DBI qw(:sql_types);
use DBD::mysql;
BEGIN {
import Amavis::Conf qw(:platform :confvars c cr ca $myhostname);
import Amavis::Util qw(do_log untaint safe_encode safe_decode);
}
sub new {
my($class,$conn,$msginfo) = @_;
my($self) = bless {}, $class;
$self; # returning an object activates further callbacks,
# returning undef disables them
}

do_log($ll,"CUSTOM: checks: status: $status");
if ($status =~ /ST_CONTENT=/) {
do_log($ll,"CUSTOM: checks: ST_CONTENT");
$msginfo->add_contents_category(CC_CLEAN,7);
for my $r (@{$msginfo->per_recip_data}) {
$r->add_contents_category(CC_CLEAN,7);
}
}


this doesn't: (it correctly logs mail_id and partition tag) but db
connection doesn't seem to work.

sub after_send {
my($self,$conn,$msginfo) = @_;
my($conn_h) = $self->{'conn_h'};
my($mail_id) = $msginfo->mail_id;
my($partition_tag) = $msginfo->partition_tag;
my($content) = $msginfo->is_in_contents_category(CC_CLEAN,7);
do_log(0, "CUSTOM: after_send, content=$content, mail_id= $mail_id,
partag=$partition_tag");
if ($content) {
my($query) = "update msgs set content=? where mail_id=? and
partition_tag=?";
$conn_h->begin_work_nontransaction;
$conn_h->execute($query,"T",$mail_id,$partition_tag);
}
}


Feb 2 15:58:55 fl amavis[59162]: (59162-01) CUSTOM: checks: status:
AWL=-0.353,BAYES_50=0.001,RDNS_NONE=1.1, ST_CONTENT=0.01
Feb 2 15:58:55 fl amavis[59162]: (59162-01) CUSTOM: checks: ST_CONTENT
Feb 2 15:58:55 fl amavis[59162]: (59162-01) CUSTOM: before_send
Feb 2 15:58:55 fl amavis[59162]: (59162-01) CUSTOM: CONTENT = 1
Feb 2 15:58:55 fl amavis[59162]: (59162-01) CUSTOM: after_send,
content=1, mail_id= Jz6Z224jqIxF, partag=0
Feb 2 15:58:55 fl amavis[59162]: (59162-01) (!)custom after_send error:
Can't call method "begin_work_nontransaction" on an undefined value at
/var/amavis/etc/amavisd.conf line 123, <GEN19> line 66.

line 123 is:
$conn_h->begin_work_nontransaction;
--
Michael Scheidell, CTO
Phone: 561-999-5000, x 1259
*| *SECNAP Network Security Corporation
* Certified SNORT Integrator
* King of Spam Filters, SC Magazine 2008
* Information Security Award 2008, Info Security Products Guide
* CRN Magazine Top 40 Emerging Security Vendors
* Finalist 2009 Network Products Guide Hot Companies

_________________________________________________________________________
This email has been scanned and certified safe by SpammerTrap(r).
For Information please see http://www.secnap.com/products/spammertrap/
_________________________________________________________________________

------------------------------------------------------------------------------
Create and Deploy Rich Internet Apps outside the browser with Adobe(R)AIR(TM)
software. With Adobe AIR, Ajax developers can use existing skills and code to
build responsive, highly engaging applications that combine the power of local
resources and data with the reach of the web. Download the Adobe AIR SDK and
Ajax docs to start building applications today-http://p.sf.net/sfu/adobe-com
_______________________________________________
AMaViS-user mailing list
AMaViS-***@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/amavis-user
AMaViS-FAQ:http://www.amavis.org/amavis-faq.php3
AMaViS-HowTos:http://www.amavis.org/howto/
Mark Martinec
2009-02-04 16:40:13 UTC
Permalink
Michael,
Post by Michael Scheidell
I want to set msgs.content = 'T' if CC_CLEAN,7. do I do it in
custom-after_send() or custom_end()?
It is currently hard-coded in sub save_info_final, you'll need
to adjust it there, sorry:

my($content_type) = $msginfo->setting_by_contents_category({
CC_VIRUS,'V', CC_BANNED,'B', CC_SPAM,'S', CC_SPAMMY,'s',
CC_BADH.",2",'M', CC_BADH,'H', CC_OVERSIZED,'O',
CC_CLEAN,'C', CC_CATCHALL,'?'});
Post by Michael Scheidell
is the (persistant?) db connection still available after_send() or end()?
Yes, should be.
...
Post by Michael Scheidell
this doesn't: (it correctly logs mail_id and partition tag) but db
connection doesn't seem to work.
sub after_send {
my($conn_h) = $self->{'conn_h'};
my($mail_id) = $msginfo->mail_id;
my($partition_tag) = $msginfo->partition_tag;
my($content) = $msginfo->is_in_contents_category(CC_CLEAN,7);
if ($content) {
my($query) = "update msgs set content=? where mail_id=? and
partition_tag=?";
$conn_h->begin_work_nontransaction;
$conn_h->execute($query,"T",$mail_id,$partition_tag);
}
}
Can't call method "begin_work_nontransaction" on an undefined value at
/var/amavis/etc/amavisd.conf line 123, <GEN19> line 66.
$conn_h->begin_work_nontransaction;
The $self in custom hooks is your own object, often just
an empty blessed hash. You can store there what you need.
Post by Michael Scheidell
my($conn_h) = $self->{'conn_h'};
This is not the persistent connection as used in the rest of
amavisd. The example amavisd-custom.conf shows how to make
you own SQL connection independent of amavisd SQL use:

sub new {
my($class,$conn,$msginfo) = @_;
my($self) = bless {}, $class;
my($conn_h) = Amavis::Out::SQL::Connection->new(
['DBI:mysql:database=user_presence;host=127.0.0.1', 'user1', 'passwd1'] );
$self->{'conn_h'} = $conn_h;
$self;
}


If you need access to the SQL connection as used by amavisd
for lookups or SQL logging, you must go through objects
$sql_policy or $sql_storage - see module Amavis::Lookup::SQL

Mark

------------------------------------------------------------------------------
Create and Deploy Rich Internet Apps outside the browser with Adobe(R)AIR(TM)
software. With Adobe AIR, Ajax developers can use existing skills and code to
build responsive, highly engaging applications that combine the power of local
resources and data with the reach of the web. Download the Adobe AIR SDK and
Ajax docs to start building applications today-http://p.sf.net/sfu/adobe-com
_______________________________________________
AMaViS-user mailing list
AMaViS-***@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/amavis-user
AMaViS-FAQ:http://www.amavis.org/amavis-faq.php3
AMaViS-HowTos:http://www.amavis.org/howto/
Michael Scheidell
2009-02-04 16:58:10 UTC
Permalink
Post by Mark Martinec
Michael,
It is currently hard-coded in sub save_info_final, you'll need
my($content_type) = $msginfo->setting_by_contents_category({
CC_VIRUS,'V', CC_BANNED,'B', CC_SPAM,'S', CC_SPAMMY,'s',
CC_BADH.",2",'M', CC_BADH,'H', CC_OVERSIZED,'O',
CC_CLEAN,'C', CC_CATCHALL,'?'});
I did it like this in custom-mail_done():

if ($content) {
my($query) = "update msgs set content=? where mail_id=? and
partition_tag=? and content not in('V','B')";
$conn_h->begin_work_nontransaction;
$conn_h->execute($query,"T",$mail_id,$partition_tag);
}
Post by Mark Martinec
Post by Michael Scheidell
is the (persistant?) db connection still available after_send() or end()?
Yes, should be.
[....]
Post by Mark Martinec
This is not the persistent connection as used in the rest of
amavisd. The example amavisd-custom.conf shows how to make
If you need access to the SQL connection as used by amavisd
for lookups or SQL logging, you must go through objects
$sql_policy or $sql_storage - see module Amavis::Lookup::SQL
So in custom-new() it can't 'grab' the global conn-h? In wrong name/stack
space?

Ps, had an inspiration on per-user or per-domain bayes for anyone who still
asks for that.

Disable bayes in standard local.cf. During custom-checks(), look for
policy/direction, 'local=y' (so you can tell if you want to use the sender
or the recipient domain). Lookup the user# if using per user, or domain# if
using domain and index into bayes_vars, then set ENV, per user!

During

for my $r (@{$msginfo->per_recip_data}) {
Set ENV (user),
Call SA->new() checks() with JUST THE BAYES CF (yes, you can call SA with
just ONE plugin/cf))
Add back in to scores, modify 'status' headers.
}

Just a thought.
--
Michael Scheidell, CTO
Post by Mark Martinec
|SECNAP Network Security
Finalist 2009 Network Products Guide Hot Companies
FreeBSD SpamAssassin Ports maintainer


_________________________________________________________________________
This email has been scanned and certified safe by SpammerTrap(r).
For Information please see http://www.secnap.com/products/spammertrap/
_________________________________________________________________________

------------------------------------------------------------------------------
Create and Deploy Rich Internet Apps outside the browser with Adobe(R)AIR(TM)
software. With Adobe AIR, Ajax developers can use existing skills and code to
build responsive, highly engaging applications that combine the power of local
resources and data with the reach of the web. Download the Adobe AIR SDK and
Ajax docs to start building applications today-http://p.sf.net/sfu/adobe-com
_______________________________________________
AMaViS-user mailing list
AMaViS-***@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/amavis-user
AMaViS-FAQ:http://www.amavis.org/amavis-faq.php3
AMaViS-HowTos:http://www.amavis.org/howto/
Continue reading on narkive:
Loading...