Need help with .htaccess and robots files. Kind of urgent.

tonoshi · 08-23-2010 03:00 AM

Im really struggling with my .htaccess and robots settings for my forum.

I have lost all my listings in google and have 2 pages listed out of 300+

I have been tweeking things in the 2 files for various things and now im lost.

Would someone be able to review the files if i posted them? Please, I really need some help. It is for a vb forum btw.

This is what changes i have made from lastnight to try and get this working

.htaccess

Code:

Order Allow,Deny
Deny from 178.162.161.42
# Deny from 212.113.32.      # Ukraine
# Deny from 212.113.33.      # Ukraine
# Deny from 212.113.34.      # Ukraine
# Deny from 212.113.35.      # Ukraine
# Deny from 212.113.36.      # Ukraine
# Deny from 123.152.         # China Unicom Zhejiang province network
# Deny from 123.153.         # China Unicom Zhejiang province network
# Deny from 123.154.         # China Unicom Zhejiang province network
# Deny from 123.155.         # China Unicom Zhejiang province network
# Deny from 123.156.         # China Unicom Zhejiang province network
# Deny from 123.157.         # China Unicom Zhejiang province network
# Deny from 123.158.         # China Unicom Zhejiang province network
# Deny from 123.159.         # China Unicom Zhejiang province network
# Deny from 22.224.          # USA
# Deny from 81.92.151.       # Czech Republic
# Deny from 115.             # China
# Deny from 122.224.135.     # China
# Deny from 218.61.          # China
# Deny from 218.60.          # China
# Deny from 114.           # China
# Deny from 30.58.205.       # Spain Madrid
# Deny from 217.65.          # Hungury
# Deny from 200.75.          # America and Africa
# Deny from 200.161.         # America and Africa
# Deny from 121.176.         # Korea Republic
# Deny from 78.94.           # Germany
# Deny from 85.62.           # Spain 
# Deny from 188.228.         # Denmark Skanderborg - 188.228.33.237
# Deny from 173.236.         # United States Chicago Singlehop Inc 
Allow from all

# ErrorDocument 403 http://errors.redirectme.net/error403.htm
# ErrorDocument 400 http://errors.redirectme.net/error403.htm

DirectoryIndex index.php

RewriteEngine on
# RewriteRule ^index.php /$1 [R=301,L]
# RewriteRule ^index.html /$1 [R=301,L]
RewriteRule ^forum.php /$1 [R=301,L]

RewriteBase /
RewriteCond %{REQUEST_FILENAME} -s [OR]
RewriteCond %{REQUEST_FILENAME} -l [OR]
RewriteCond %{REQUEST_FILENAME} -d

RewriteRule ^.*$ - [NC,L]
RewriteRule ^threads/.* showthread.php [QSA]
RewriteRule ^forums/.* forumdisplay.php [QSA]
RewriteRule ^members/.* member.php [QSA]
RewriteRule ^blogs/.* blog.php [QSA]
ReWriteRule ^entries/.* entry.php [QSA]
RewriteRule ^list/([^/]*/)([0-9]+) list.php?r=$1$2&%{QUERY_STRING}
RewriteRule ^content/(.*) content.php?r=$1&%{QUERY_STRING}
RewriteRule ^widget/config/([0-9]+) widget.php?r=config/$1&%{QUERY_STRING}
RewriteRule ^$ $1.php?r=$2 [QSA]

My robots.txt

Code:

User-agent: *
Disallow: /tags.php
Disallow: /index.php
Disallow: /arcade.php
Disallow: /editpost.php
Disallow: /inlinemod.php
Disallow: /member.php
Disallow: /memberlist.php
Disallow: /newreply.php
Disallow: /payments.php
Disallow: /printthread.php
Disallow: /private.php
Disallow: /profile.php
Disallow: /faq.php
Disallow: /report.php
Disallow: /search.php
Disallow: /sendmessage.php
Disallow: /showpost.php
Disallow: /usercp.php
Disallow: /usernote.php
Disallow: /chat/
Disallow: /arcade/
Disallow: /ajax.php

Sitemap: http://mysite.com/vbulletin_sitemap_index.xml.gz

Still only 2 pages index, no errors in WMT. Any help would be gratefully recevied.

This was the robots i was running

Code:

User-agent: *
Disallow: /*arcade.php
Disallow: /index.php
Disallow: /*external.php
Disallow: /*attachment.php
Disallow: /*ajax.php
Disallow: /*misc.php
Disallow: /*calendar.php
Disallow: /archive/
Disallow: /*faq.php
Disallow: /*editpost.php
Disallow: /*inlinemod.php
Disallow: /*member.php
Disallow: /*memberlist.php
Disallow: /*newreply.php
Disallow: /*newthread.php
Disallow: /*payments.php
Disallow: /*printthread.php
Disallow: /*private.php
Disallow: /*profile.php
Disallow: /*report.php
Disallow: /*search.php
Disallow: /*sendmessage.php
Disallow: /*usercp.php
Disallow: /*usernote.php
Disallow: /cpstyles/
Disallow: /arcade/
Disallow: /chat/
Disallow: /customavatars/
Disallow: /customcode/
Disallow: /customprofilepics/
Disallow: /images/file_system/
Disallow: /signaturepics/
Disallow: /*tags.php
Disallow: /*.sql$
Disallow: /*.tpl$
Disallow: /blog.php?1-TazDevilLooney&blogcategoryid=1
Disallow: /blog.php?1-TazDevilLooney&tag=how%20to
Disallow: /blog.php?cp=1
Disallow: /blog.php?b=1&goto=next
Disallow: /showthread.php?6-Magic-test&p=0
Disallow: /threadtag.php?do=manage&contenttype=vBBlog_BlogEntry&contentid=1
Disallow: /blogs/1-TazDevilLooney?blogcategoryid=1
Disallow: /blogs/1-TazDevilLooney?m=4&y=2010
Disallow: /blog.php?1-Taz-s-Blog-Log&m=5&y=2010
Disallow: /*group.php?do=viewmembers
Disallow: /content/10-Flash-Game
Disallow: /content.php?13
Disallow: /content.php?18-GunSlinger
Disallow: /list.php?day/1271545201
Disallow: /content.php?10-Flash-Game
Disallow: /content.php?14
Disallow: /content.php?17-Nuns-on-the-run
Disallow: /content.php?15
Disallow: /content.php?16-solitairious

Sitemap: http://mysite.com/vbulletin_sitemap_index.xml.gz

***ZiNgA BuRgA*** · 08-23-2010 07:53 AM

.htaccess and robots.txt won't stop Google indexing your pages unless you've actually blocked it with them.

.htaccess is unlikely the problem, since you can access the website fine, Google should too.
For robots.txt, you can check with Google's webmaster tools to see if it's not accessing the pages you want it to.

Threaded Mode \| Linear Mode Standard Tools
Forum Jump: View a Printable Version Send this Thread to a Friend Subscribe to this thread