In terms of code or forum settings?
I don't really know that much to be honest. I've generally been more interested in optimising for speed, since the memory is only going to be used for however long the script takes to run, and as long as you're not doing something silly, you won't be consuming too much (and PHP will kill your script if you use too much anyway). I think MyBB's memory usage thing at the bottom only measures peak usage too, which may only be allocated at a tiny part during script execution.
PHP, being such a high level scripting language, is somewhat difficult to predict how much memory it will allocate; although I guess you could say speed is difficult to predict, but memory usage would be even harder to predict.
Apart from certain admin functions, I think the biggest memory users in MyBB, would potentially be the attachment download script. The script reads the entire attachment file into memory and it will stay in memory until the script finishes running (ie when user finishes downloading (not quite, due to potential webserver buffering)). So if a user is trying to download a 2MB attachment, MyBB reads the entire file into memory, consuming 2MB data memory plus any PHP string overheads (also, plus MyBB's core overheads) and this will stay in memory until the user nearly finishes downloading the file.
So to avoid that, I guess, don't allow really big attachments, or do a code edit to attachment.php, changing
to
(XThreads attachment downloads do not suffer from this issue)
There are potentially a number of other places where a lot of memory could be used, for example, using a remote (URL) avatar, if a user doesn't like you and sticks like a link to a 4GB file there or something like that.
As for coding, I guess it depends on the number of variables and size of each you have, resources allocated, and amount of code loaded.