wikidot-forum-dump
Script is partially based on 2stacks by bluesoul: https://github.com/scuttle/2stacks
To dump a Wiki's forum, edit config.py and put the required Wiki name (default — scp-wiki, which is the SCP EN community).
If your forum is too large, you may edit threads parameter in config.py, however make sure you don't overload Wikidot with requests — no one knows what that may result in.
Then make sure you have the following Python packages:
beautifulsoup4
Run python . to start dumping.
Note: incremental dump (i.e. updating existing categories or threads) is not supported.
However, if the process is interrupted at any point, it can be seamlessly resumed per-category and per-thread.
The dump will be stored under the following structure:
dump/
categories.json -- contains group and category names and IDs
categories/
.json -- contains list of threads for each category by ID
threads/
.json -- contains each thread with all posts and replies
Example of a group+categories record (from SCP-EN):
{
"title": "Site Announcements and Proposals",
"description": "Announce new pages, suggest policy, and interact with new site members.",
"categories": [
{
"title": "Sitewide Announcements",
"description": "Announcement of any sitewide changes or events. For usage by both staff and non-staff.",
"id": 1113520
},
{
"title": "Page Announcements",
"description": "Announce posting of new pages and deletion of old pages. Authors, please use the collective SCP, Tale, GOI Entry, and Update threads for new works.",
"id": 7409511
},
{
"title": "Proposals And Policy",
"description": "What can we do to improve the site? Ask any questions you may have regarding site structure and policy.",
"id": 51015
},
{
"title": "Introductions",
"description": "New to the site? Introduce yourself and meet other site members here.",
"id": 72352
}
]
}
Example of a thread record (also from SCP-EN):
Currently, we're having some Wikidot problems with the Master Admin position. We've moved the Master Admin position to a different account (under Mann's control) in the meantime, but there may be problems for a few days while we sort things out.
\nThe thing most prominently affected by this will be uploaded files. Until the issue is resolved, please host images on an external hosting site like Imgur. After image uploading has resumed functioning, Staff will reupload the images on the mainsite manually. This will affect the sandbox as well.
\nHelen (our IRC bot) will also be temporarily down until the problem has been resolved. This also means some Helen-adjacent functions may not be functional, or may experience issues.
\nWe thank you for your cooperation during this. If you have any questions about the above or any other problems you come across, feel free to ask in this thread or message a staff member. Thank you.
", "replies": [ { "user": { "avatar": "http://www.wikidot.com/avatar.php?userid=3695324&size=small×tamp=1640941899", "name": "CephalopodStevenson", "id": 3695324 }, "date": 1640903834, "title": "Re: Technical Issues Announcement", "content": "So, I’ll preface by saying that I have very little knowledge of how wikidot works. I hope somebody can enlighten me here and explain what any of this means. Why does a master admin problem affect media files? What exactly is the problem? Who was controlling the master admin account before, and is this an issue with said user or wikidot itself?
\nAlso, “most prominently affected” suggests that there may be other concerns, right? I’m guessing this means we should be backing up our content regardless of media inserts. Again, I’m not super knowledgeable on any of this, so if someone could clarify a bit, I’d appreciate it.
", "replies": [ { "user": { "avatar": "http://www.wikidot.com/avatar.php?userid=2005044&size=small×tamp=1640941899", "name": "Decibelles", "id": 2005044 }, "date": 1640905152, "title": "Re: Technical Issues Announcement", "replies": [], "content": "
\n
Why does a master admin problem affect media files? What exactly is the problem? Who was controlling the master admin account before, and is this an issue with said user or wikidot itself?
\n
\n
This one's a bit of a tricky one to explain, as the answers are all interconnected. The Master Admin before was Mann. Mann has a Pro account. Wikidot's subscription plans allow a site to have a bunch of features that you cannot have if you make a site with a Free account. Among these features is an upgrade in storage space, IE, how much you can upload to a site. In addition to this, you can also buy more storage as well.
\nThe master admin status presumably has shifted from Mann to another account, still under Mann's control. However, this other account presumably does not have Pro. So therefore, storage limits will have naturally been hit. I can't pretend to know what the issue was that resulted in the transfer of status, but this would end up resulting in storage now being at a premium until the situation can be resolved.
\n
\n
I’m guessing this means we should be backing up our content regardless of media inserts.
\n
\n
I'm not sure what anything else affected could be, but it is 100% always a good idea to back up your content from this site as best as you are able to.
" }, { "user": { "avatar": "http://www.wikidot.com/avatar.php?userid=2199269&size=small×tamp=1640941899", "name": "Yossipossi", "id": 2199269 }, "date": 1640905211, "title": "Re: Technical Issues Announcement", "content": "\n
\n ", "replies": [ { "user": { "avatar": "http://www.wikidot.com/avatar.php?userid=3695324&size=small×tamp=1640941899", "name": "CephalopodStevenson", "id": 3695324 }, "date": 1640905510, "title": "Re: Technical Issues Announcement", "replies": [], "content": "
Thanks both of you for the informative responses. I feel I have a much better understanding of the situation now.
" } ] } ] } ] } ] }">{
"breadcrumbs": [
"Forum",
"Site Announcements and Proposals / Sitewide Announcements",
"Technical Issues Announcement"
],
"category": 1113520,
"base_page_id": null,
"description": "An announcement regarding some Wikidot technical difficulties and mitigations, with particular reference to file uploads.",
"user": {
"avatar": "http://www.wikidot.com/avatar.php?userid=3075960&size=small×tamp=1640941899",
"name": "stormbreath",
"id": 3075960
},
"date": 1640899718,
"posts": [
{
"user": {
"avatar": "http://www.wikidot.com/avatar.php?userid=3075960&size=small×tamp=1640941899",
"name": "stormbreath",
"id": 3075960
},
"date": 1640899718,
"title": "Technical Issues Announcement",
"content": "Hey everyone.
\nCurrently, we're having some Wikidot problems with the Master Admin position. We've moved the Master Admin position to a different account (under Mann's control) in the meantime, but there may be problems for a few days while we sort things out.
\nThe thing most prominently affected by this will be uploaded files. Until the issue is resolved, please host images on an external hosting site like Imgur. After image uploading has resumed functioning, Staff will reupload the images on the mainsite manually. This will affect the sandbox as well.
\nHelen (our IRC bot) will also be temporarily down until the problem has been resolved. This also means some Helen-adjacent functions may not be functional, or may experience issues.
\nWe thank you for your cooperation during this. If you have any questions about the above or any other problems you come across, feel free to ask in this thread or message a staff member. Thank you.
",
"replies": [
{
"user": {
"avatar": "http://www.wikidot.com/avatar.php?userid=3695324&size=small×tamp=1640941899",
"name": "CephalopodStevenson",
"id": 3695324
},
"date": 1640903834,
"title": "Re: Technical Issues Announcement",
"content": "So, I’ll preface by saying that I have very little knowledge of how wikidot works. I hope somebody can enlighten me here and explain what any of this means. Why does a master admin problem affect media files? What exactly is the problem? Who was controlling the master admin account before, and is this an issue with said user or wikidot itself?
\nAlso, “most prominently affected” suggests that there may be other concerns, right? I’m guessing this means we should be backing up our content regardless of media inserts. Again, I’m not super knowledgeable on any of this, so if someone could clarify a bit, I’d appreciate it.
",
"replies": [
{
"user": {
"avatar": "http://www.wikidot.com/avatar.php?userid=2005044&size=small×tamp=1640941899",
"name": "Decibelles",
"id": 2005044
},
"date": 1640905152,
"title": "Re: Technical Issues Announcement",
"replies": [],
"content": "
\n
Why does a master admin problem affect media files? What exactly is the problem? Who was controlling the master admin account before, and is this an issue with said user or wikidot itself?
\n
\nThis one's a bit of a tricky one to explain, as the answers are all interconnected. The Master Admin before was Mann. Mann has a Pro account. Wikidot's subscription plans allow a site to have a bunch of features that you cannot have if you make a site with a Free account. Among these features is an upgrade in storage space, IE, how much you can upload to a site. In addition to this, you can also buy more storage as well.
\nThe master admin status presumably has shifted from Mann to another account, still under Mann's control. However, this other account presumably does not have Pro. So therefore, storage limits will have naturally been hit. I can't pretend to know what the issue was that resulted in the transfer of status, but this would end up resulting in storage now being at a premium until the situation can be resolved.
\n
\n
I’m guessing this means we should be backing up our content regardless of media inserts.
\n
\nI'm not sure what anything else affected could be, but it is 100% always a good idea to back up your content from this site as best as you are able to.
"
},
{
"user": {
"avatar": "http://www.wikidot.com/avatar.php?userid=2199269&size=small×tamp=1640941899",
"name": "Yossipossi",
"id": 2199269
},
"date": 1640905211,
"title": "Re: Technical Issues Announcement",
"content": "
\n
\n
\n
\n
\n
\n
\n
\n ",
"replies": [
{
"user": {
"avatar": "http://www.wikidot.com/avatar.php?userid=3695324&size=small×tamp=1640941899",
"name": "CephalopodStevenson",
"id": 3695324
},
"date": 1640905510,
"title": "Re: Technical Issues Announcement",
"replies": [],
"content": "Thanks both of you for the informative responses. I feel I have a much better understanding of the situation now.
"
}
]
}
]
}
]
}
]
}
Note: contrarily to what it says, base_page_id is not an ID, but a slug (e.g. scp-173).
This exists only for page discussion threads.
6 Dec 7, 2021
14 Dec 22, 2022
1.5k Jan 5, 2023
79 Nov 11, 2022
0 Jan 22, 2022
235 Dec 29, 2022
51 Dec 1, 2021
3 Jan 9, 2022
4 Aug 22, 2022
1 Dec 14, 2022
68 Dec 7, 2022
4 Mar 7, 2022
4 Apr 24, 2022
3 Feb 2, 2022
86 Jan 3, 2023
1 Oct 14, 2021
9 Dec 2, 2022
44 Dec 29, 2022
150 Nov 23, 2022