Sitemap for Google - PHPBB 3 Mod

[Liv]

Modification Name: PHPBB Site Map for Google Mod

Modification Version:: 18 (22 Mar 2013) Validated for 3.0.11 and prior.
Author: L. Jones

Modification Description::

PHPBB SiteMap for Google Mod produces an XML (by redirect) or pseudo XML compliant link tree for use with various sitemap submission softwares such as Google's Webmaster tools sitemap submission service or Yahoo's Site Explorer. The file is tiny, and simple to install.

Features:

Simple installation, "upload and go" if your forum is in the root domain. Simply upload the PHP file (sitemap.php) and add the link to your index page. Otherwise, modify the domain variables in the top of the PHP file and off you go. The site-map provides real-time 24 hour data on topics to Search Engines down to the second.

Screenshots:

Phpbb Sitemap Mod

Demo URL: sitemap.php

Modification Download:

Google Sitemap for PHPBB v18.zip

(2.5 KiB) Downloaded 156 times

Site Map Install Instructions:

1) Download the above sitemap in Zip form and decompress on your computer. (above)

2) If Your PHPBB3 Forum is installed in the root folder, for instance at http://greensboring.com then skip "a"

a) modify the $folder or $subdomain variable to match your forum. An example is given at the top of the PHP code.
3) Upload to your forum root folder. (where PHPBB's index.php is)
4) Access your PHPBB's index.php from your forum root folder and open it for editing, find:


Code: Select all

'TOTAL_POSTS'   => sprintf($user->lang[$l_total_post_s], $total_posts),


After that add:


Code: Select all

'SITE_MAP' => '<a target="_blank" href="sitemap.php" title="sitemap" rel="alternate" type="application/rss+xml">SiteMap</a> by <a target="_blank" href="http://www.livjones.com">Liv</a>',


5) Go into your administrator control panel -> styles -> template -> edit -> "index_body.html" ->
find in template:


Code: Select all

<!-- INCLUDE overall_footer.html -->



before it add:
(Running a SubSilver or a custom theme? No problem... just add it where you feel most comfortable in the template)


Code: Select all

{SITE_MAP}




Click submit, and you're almost done...

6) Submit to Google, at Google Sitemaps.


This sitemap also works for Bing, (and others) etc. If you wish to submit it you may do so at Bing Webmaster Tools (formerly Yahoo Site Explorer)

7) Go get a lovely pint of Guinness and drink because you are officially done!

EXTRA CREDIT!!!! (optional)
Advanced users may wish to add to their .htaccess file the following command to make it fully xml compliant. This really has no effect on anything unless a random bot comes along looking for a sitemap.xml. I would honestly just recommend skipping this, unless it's important to you to have it as sitemap.xml

add (or create) in .htaccess file


Code: Select all

Options +FollowSymlinks
RewriteEngine on
RewriteRule sitemap.xml sitemap.php [NC]

PHPBB3 Sitemap Change Log:

Version: 18 (and prior) Changes:
-Split Sql requests into forums for use in large forums (v18)
-Added NOINDEX X-tag header (v17)
-Fixed cURL extension issue, when extension is not installed.
-Added Gzip compression which greatly increases the speed.
-Refined SQL SELECTs to speed reduce resource overhead.
-Updated code to handle parent forums and "1969" issue with Google sitemaps.
-Depreciated file_get_contents() in favor of Curl. (hat-tip to Marcel)
-Added Limits to the number of URLs, set default to Google's 50,000 Max.
-Fixed glitch where forums with subforums wouldn't display topics or forums.
-Added a echo statement for path if script die()'s because of faulty path.
-XML Compliant Header added.
-Corrected last mod tags format for W3c standards.
-Added LastMod tags to sitemap with data pulled from phpbb3 database.
-Only displays non-private content of both forums and topics.
-Non-Indexable forums have been removed from the sitemap.
-Changed Mysql_fetch_array to sql_fetchrow per Dave Turner to solve database compatibility issues.
-Add subdomain variable for ease. Allows user to change domain if not set by server.
-Pulls PHPBB prefix from config.php
-removed the need to state domain and path. Made it more intuitive by only needing to state folder of PHPBB install.
-Moved Forums to be listed last.
-Changed viewtopic posts to only those that are approved.
-List latest topics first rather than last.

[Sj0din]

Hi.

Im using php3 seo mod and my urls are like phpbb3.com/a-new-topic/

in your sitemap mod the urls will go /viewtopic.php?f=6&t=3329

How can I change this?

[Liv]

You can use the mod in the form it is if the pages are reachable by both URLs and you use a canacolization tag on your pages....

else you can revert back to the original PHPBB url architecture...

else I'd be glad to write you a custom version for a few hundred dollars.

[snow]

Liv is there a way to remove certain forums from being generated in the sitemap?
And since this is constantly updated, do I have to resubmit everyday to Google?
How many times should I resubmit the sitemap?

Regards,
snow

[Liv]

Liv is there a way to remove certain forums from being generated in the sitemap?
And since this is constantly updated, do I have to resubmit everyday to Google?
How many times should I resubmit the sitemap?

Regards,
snow

Yes, under forum administration in your ACP change this option to "no" for the forums you don't want included:

Code: Select all

Enable search indexing:
If set to yes posts made to this forum will be indexed for searching.

And... No... you should never have to re-submit the sitemap after the first time, Google is intelligent enough to check the sitemap prior to crawls. If your site is popular enough you'll see it automatically updated with crawl dates every so many days. The only time you would want to resubmit the sitemap is if your site suffered a pro-longed outage resulting in Google not finding the sitemap for prolonged period of time.

[newbie111]

I'm pretty new to all this, so please forgive me if this is an ignorant question.

My site uses the subsilver2 template. When I came to edit index_body.html I could not find the

<!-- INCLUDE overall_footer.html -->

line of code, which does exist in the prosilver template.

Is there a way to use this mod with the subsilver2 template?

[Liv]

That's okay... basically just add

Code: Select all

{SITE_MAP}

...anywhere at the bottom of the template and you'll be fine. You can even move it where you want.... it just adds a link to your sitemap so crawlers can find it, and a courtesy link to my site for all my hardwork. That's all.

As long as the info appears on your index page you'll be fine.

[newbie111]

Thanks heaps, it works just great

[jv]

Hello,

Will this work with SEF urls?

Thanks!

[Liv]

Indirectly if you have the proper redirects.... Google should be able to find it, but unfortunately this does not alter PHPBB's URL structure on its own.

[Liv]

It appears fine for me. Google will show them in your sitemap dashboard differently but unless you`re getting an actual warning you`re fine. Just looking at your sitemap remotely shows they`re parsing correctly.

[Liv]

No, no problem... sitemap doesn't use any $_GET variables.... so it doesn't matter.

[Liv]

It didn't for me, but I'd assume if it did at one time, it's a incorrect re-dirrect in your .htaccess. Check that.

[Liv]

If that's your .htaccess then that's not it....

I can't imagine anything that would cause a redirection to the sitemap other than a "redirect". There must be some sort of conflict with the "portal".

I'd suggest attempting to make sure you're only making changes to the PHPBB forum software, and not the portal.

[Liv]

Hi there, how long does it take, until sitemap is generated and how often those it update?

Thank you

The sitemap is generated when the file is loaded, and provides real-time data. The only situation where this would not occur, is if you are cacheing the file, which is not done by default.

[Liv]

I am getting the red X - unsupported file format error for the url in Google Sitemaps.

What's the URL?

[nilands55]

Hello
I had no problems installing the Modification I thought, but I am getting this error

Code: Select all

Something went wrong, check your path first: http://help.nilandsplace.com/campingforum/
We could not find an author citation link: http://livjones.com - Please refer to this webpage for full installation instructions:http://greensboring.com/viewtopic.php?f=23&t=1563 - Thanks!

The URL is correct. Did I miss something? I really don't know PHP coding so I have no idea what is wrong
James Niland
Laurinburg NC

[Liv]

I don't see the URL to the author on your website.... It's possible you modified the pro_silver theme rather than the one you're using. Add a link to your theme... and it should work.

[BlancBim]

thank you for your fast answer, I'm not really good at this (or even at reading english ) so here is what I did...

I unzipped the new file, copyed the file "sitemap.php" past and overwrite the old one in the forum's directory.
then went to the index page of the forum and hit the sitemap link

first I got this error:

Code: Select all

Something went wrong, check your path first: http://tls3d.fr/
We could not find an author citation link: http://www.livjones.com on your index page. - Please refer to this webpage for full installation instructions:http://greensboring.com/viewtopic.php?f=23&t=1563 - Thanks!

so I changed the line:
$folder='';
to
$folder='/forum/';

and I get this error:

Code: Select all

Something went wrong, check your path first: http://tls3d.fr/forum/
We could not find an author citation link: http://www.livjones.com on your index page. - Please refer to this webpage for full installation instructions:http://greensboring.com/viewtopic.php?f=23&t=1563 - Thanks!

it now points to the right directory but still the same error, I also tryed with the line:
$path='/forum/';
but it doesn't seem to change anything till now

[Liv]

I'm not really good at this (or even at reading english ) so here is what I did...

C'est cool! No problem.

I can only guess that cURL is still the issue. I personally believe it's to do with Gzip compression on some servers.

Here's a work-around:

Delete this in sitemap.php

Code: Select all

// Initiate Engines for countdown start

if (function_exists('curl_init')){

$file = $domainpath . 'index.' . $phpEx;
$ch = curl_init();
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_URL, $file);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, TRUE);
$fil3 = curl_exec($ch);
curl_close($ch);
$url=base64_decode('aHR0cDovL3d3dy5saXZqb25lcy5jb20=');
if (strpos($fil3,$url) === FALSE)
 {echo 'Something went wrong, check your path first: '.$domainpath.'<br>';
die(base64_decode('V2UgY291bGQgbm90IGZpbmQgYW4gYXV0aG9yIGNpdGF0aW9uIGxpbms6IDxhIGhyZWY9Imh0dHA6Ly93d3cubGl2am9uZXMuY29tIj5odHRwOi8vd3d3LmxpdmpvbmVzLmNvbTwvYT4gb24geW91ciBpbmRleCBwYWdlLiAtIFBsZWFzZSByZWZlciB0byB0aGlzIHdlYnBhZ2UgZm9yIGZ1bGwgaW5zdGFsbGF0aW9uIGluc3RydWN0aW9uczo8YSBocmVmPSJodHRwOi8vdGhlc2VjdWxhcml0eS5jb20vdmlld3RvcGljLnBocD9mPTIzJnQ9MTU2MyI+aHR0cDovL3RoZXNlY3VsYXJpdHkuY29tL3ZpZXd0b3BpYy5waHA/Zj0yMyZ0PTE1NjM8L2E+IC0gVGhhbmtzIQ=='));}

}


Save, and re-upload. It should work!

[htmanning]

It works, thanks! I've used it on 3 forums, and I'm getting errors on the third forum so I have to conclude there is some entry in my database that is causing the problem.

I'm getting this message:
"error on line 21621 at column 16: Extra content at the end of the document"

It just stops at the end of a line. If you look at the source it shows a closing URL tag so that entry finished correctly. Looking at the next topic_id number down in the database doesn't show anything that I find odd.

Looking at the source there is no closing urlset tag. The sitemap just ends. I'm stumped.

I'm also getting the 1969 date error on about 12 entries, because those topics exist in the database but have no text in them. I can simply delete those.

I hope someone can help.

Thanks.

Tom

[Liv]

If you have to many URLs that your server stops responding on the script, you can definitely adjust the number of URLs to a more reasonable level. This feature was added a few versions ago, so assuming you have the latest, you'd simply adjust this parameter:

Code: Select all

$urls=50000;

(Default is 50,000)

As far as the first question, I'm not sure how you have topics in the database but are saying the topic doesn't exist. There's no real option to do that in PHPBB (to my knowledge [if there is I'd be glad to add that into a future version, but I don't see it in the tables]).

Now if the posts are in a particular forum (f=) which is private, then it's possible you could get the error because the topic wouldn't exist for search engines.

In that case you'd simply make sure that in your control panel, under the private forums you've set ENABLE SEARCH INDEXING to NO for those forums, and they'll automatically not appear in the sitemap.

Go to FORUMS > EDIT FORUM > GENERAL FORUM SETTINGS

Code: Select all

Enable search indexing:
If set to yes posts made to this forum will be indexed for searching.


and set it to NO for private forums.

This was also a feature that's built into the last few versions.

[htmanning]

Thanks for the quick response.

I adjusted the $urls number earlier and that made no difference. It still stops at the same line. I currently have it set to 100,000 although I also tried 70,000 and it made no difference.

I have one Announcements forum that only the admin can post for, but I have that same forum at other installs and it doesn't cause a problem. However, I did as you suggested and turned off indexing. Sitemap.php now stops at the same line number (21620), but at a different record. It was stopping at topic_id 901, but after turning off the indexing for that one forum it goes until topic_id 884.

1969 error: I seem to have a few topics in the database that were spam, and were removed by a moderator, but for some reason the title of the topic is still there (no post). So I'm wondering if maybe there is no corresponding entry in the database that would hold the body of the post.

Thanks.

[Liv]

My guess is it's still an execution time-out, by your host.... try a number like 5,000 and see what happens, then creep up from there in 5 or 10,000 increments till you max out your servers time out parameter.

The good news is that it does list the newest topics first, so it still should work wonders for you if that works.

As far as the phantom topics, there's two tables in PHPBB that topics come from (basically)... the Topics_Table and the Posts_Table.

It is possible that you have information in the Topics_Table (where we draw the data from for this sitemap) but not the Posts_Table.

The only way I can imagine this happening is by a modification or table merger sometime in the past. If this is the case, you can either choose to fix it, or ignore it, as it shouldn't really matter as far as the modification goes. Even if you get errors, it will still function for Google (though fixing the time-out issue is probably important).

I guess the first place to start is check the topics in question, and if they don't actually exist (meaning you can't see them with admin privileges), then something is up with your database, and you'll likely want to run a script to clean out those rogue entries (if you want to put that much effort into it).

If they do exist, then LMK if I missed something, because I'd definitely love to check and see what's going on.

[htmanning]

Looks like you're on to something. Anything more than $urls=7000 and it chokes.