Spamalyser

In short, Spamalyser tries to detect spam posts (not spam registrations) through a number of means and can perform some actions on detected spam.
Note that this plugin isn't exactly "complete", but should be "good enough" to use.

How is this different to other anti-spam plugins?
There are a number of other MyBB plugins which try to prevent spam.  However, I find that most of these can get rather pedantic and can easily block legitimate users with little, if any, means of them working around the block.
The aim of this plugin was to be somewhat more permissive than restrictive, as well as not have to rely on external services (although this plugin is capable of utilising information pulled from them).  This tries to deny as little as possible to legit users - for example, a solution such as blocking link posting for new users means that all users need to have a number of posts before they are able to post links, whereas Spamalyser allows this activity as long as it doesn't look spammy.

Stop Forum Spam and Fassim plugins, for example, seem to only perform blocks on user registration (before they even register, in fact).  As Spamalyser is more permissive, it only performs checks on posting (this also has the slight benefit of being able to work if guest posting is enabled, although Spamalyser isn't as effective with guests than it is with registered users).  Also, performing spam analysis on posts means that Spamalyser is able to use a greater number of inputs to judge whether a post is spam or not.
Of course, the downside is that Spamalyser will not try to do anything about spam registrations.

Can this be used with other anti-spam plugins?
Most of them yes.  I'm not sure whether this works with the Akismet plugin or not (although it's somewhat pointless to do so, because Spamalyser supports Akismet lookups).
If you use it with the Stop Forum Spam plugin, you may wish to disable SFS lookups in Spamalyser.

How does this plugin work?
When a user tries to post a new thread or reply, or edit a post, the plugin will check whether the user passes a number of thresholds (such as post count) and if so, deems the user "safe" and stops there.
If the user fails to pass the threshold test, the main Spamalyser engine kicks in and analyses the post to determine a "spam weighting" (likeliness of it being spam).  It then compares this weighting against some configurable action thresholds to decide on whether it should do anything to the post.  If the weighting meets the thresholds, Spamalyser can currently (depending on what you enable):

• Report the post,
• Unapprove the post, or
• Block the post (displays an error when user tries to submit the post)

All weighting calculations are logged and can be viewed via ACP -> Tools -> Spamalyser Log

How is the "spam weighting" calculated?
Quite a number of means, but most of the code is link analysis.  Spammers ultimately want to post links, so it seems like a good place to start.  Every link posted will add to the weighting, and links with similar keywords or to the same domain get penalised more heavily.
Spamalyser can also make some judgements based on the poster's online time and other factors, and has a number of features to attempt to reduce the number of false positives (for example, by examining the user's previous posts).
External lookups to services such as Stop Forum Spam and Akismet are also supported, and you can specify the amount of weighting to give to these services.

You can try looking in the Spamalyser settings which show all the methods used by this plugin to detect spam.

Configuring Spamalyser
Spamalyser has quite a number of options, designed to give the administrator a fair amount of control over the spam analysis process.
The defaults are relatively permissive.  From my testing, you can probably reduce the Unapprove Threshold from 10 to 5 and capture a lot more spam.
Note that, by default, external lookups are disabled for localhost installs, but enabled for non-localhost installs.

I know there are a lot of options so it may be difficult to get your head around what they all do.  You can try inspecting the log (clicking on the weighting will show how it's calculated) to see how some posts' weights are determined and perhaps get a rough idea of what some of the options do.

Disable Link Analysis
If you wish to, you can completely disable the internal link analysis routine.  To do so, set the following three settings to 0:

• Weight Per Simple Link
• Weight Per Complex Link
• Duplicate Keyword Bias
  

Akismet Note
To enable Akismet, you must enter in an API key - and you must ensure that this key is valid because Spamalyser won't check it.  Also, your server must be able to either make requests via cURL or fsockopen()

Upgrading
I'm not going to bother maintaining upgrade paths for this plugin.  This means that if you wish to upgrade to a newer version, you'll have to uninstall the old version, upload the files for the new version, then install it.
Obviously this means that you'll lose any setting changes you've made, as well as log entries, so you may wish to note down any custom settings you've set before uninstalling.  Do note that the meaning of some settings may change in the future, so refer to the changelog to ensure your custom settings have the same meaning.


Performance Issues
As this plugin performs external lookups by default, for all analysed posts, posting may be slower for posts which are examined.  Although the services queried (SFS, Akismet and Google) should have reasonably fast servers, you can choose to disable these lookups if you wish. [lookups are performed through MyBB's fetch_remote_file() function]
As for internal link analysis, the algorithm isn't exactly fast (still should be significantly faster than MyBB's post parser), but should be acceptable, especially since it's only done during posting.

Limitations

• The plugin assumes that MyCode is always enabled (so only detects [url] type links), which is probably in-line with the assumptions made by most spammers
• Google searching behaviour is somewhat erratic - it's a bit difficult to generate a representative search query from a post.  Can be tweaked, though I'm not too worried because spammers can theoretically bypass this by slightly varying their posts across forums
• Reports currently only go to the database regardless of what reporting medium you have selected
  

[Shemo]

(08-02-2011 08:03 AM)ZiNgA BuRgA Wrote:  Thanks for the replies.

(08-02-2011 05:59 AM)Shemo Wrote:  zinga, do you think this plugin should be activated once spamming becomes a problem or before it?

It doesn't matter.  This plugin doesn't clean up any existing posts, only attempts to screen new ones.

thanks, I'll try it out on 1 of my more popular forums to see how it does.

[Shemo]

spamalyser zapped its first spammer on my forum.

zinga, is it possible for the plugin to ban the user that was zapped for spamming?

[ZiNgA BuRgA]

The plugin won't auto-ban - it's somewhat designed to be more permissive, as well as give legitimate posters who somehow trip the filter a chance, so it won't go that far.
I might make a task which looks for users with high amounts of blocked posts and auto-ban them, but at the moment it won't do that.

[Shemo]

(08-07-2011 10:48 AM)ZiNgA BuRgA Wrote:  The plugin won't auto-ban - it's somewhat designed to be more permissive, as well as give legitimate posters who somehow trip the filter a chance, so it won't go that far.
I might make a task which looks for users with high amounts of blocked posts and auto-ban them, but at the moment it won't do that.

no worries.  just throwing it out there while it was in my head.

to be honest, had your plugin not been installed, the spam it did pick up would of made it through with the other anti-spam plugins I got installed.  kudos for the plugin.

ps- I've been promoting in on mybb.

[ZiNgA BuRgA]

Thanks for the feedback.
If that's the case, I'd guess that it's because the post had a lot of links?

[Shemo]

(08-07-2011 05:07 PM)ZiNgA BuRgA Wrote:  Thanks for the feedback.
If that's the case, I'd guess that it's because the post had a lot of links?

usually 2-3 links when human spammers start to spam.  your plugin caught 2 spammers; one where it automatically deleted the post and the other was reported by your plugin.  in the one it deleted automatically, there was 3 links while the one it simply reported only had 2.

[ZiNgA BuRgA]

Updated to v0.93:

• Spamalyser Log
  • fix the "filtering" option (which doesn't actually filter anything yet)
  • format usernames (makes seeing banned users easier)
  • limit messages shown on the main page to 5 lines + strip excessive consecutive newlines
  • only show the reported icon in the logs if the post was _actually_ reported (eg it might not be if duplicate reporting is disabled)
• Weighting Calculation
  • fix bug where some links may wrongly be considered twice
  • fix some auto-link matching bugs
  • don't consider [video] code as a link
  • 'keywords' in URLs now carry neutral weighting [this is a rather fundamental change in regards to keyword weighting calculations]
  • don't consider image URLs as keywords any more
• Actions/Behaviour
  • don't try to unapprove posts if it's already going to be done (eg, forum requires post moderation)
  • don't attempt to analyse posts if post is going to be rejected anyway (for example, poster didn't submit a message)
  • add option to disable blocking post edits; this option is enabled by default
• Settings
  • add some numerical examples to settings
  • minor tweaks to some settings' defaults
• Other
  • fix bug where IP wouldn't be picked up correctly for a post edit
  • reduce SFS cache expiry time from 24 hours to 6 hours
  • fix SQL error in task

To upgrade, uninstall your current installation (noting any changed settings), upload the new files, then reinstall.

[x-Treme]

Thanks for the update ZB. I have been pluggin this plugin on every site I visit. To date, it's the only one that blocks spam from appearing on the forum at all. It has done a marvelous job so far and I know it will continue to do so.

[RateU]

Thanks for the update, Yumi.

[blakefire]

I am missing this one image.

action_.gif