Getting all comments from workshop projects in Steam - Part 1
Now, this is podracing.
What
The steam workshop is a Valve provided modding hub [here(https://steamcommunity.com/workshop/)]. Games on Steam can enable the workshop to allow users to easily upload and download mods for games.
This was traditionnaly a role that third party services like Nexus Mods would provide. But it requires integration with mod managers or a manual management of the mods, which is pretty cumbersome.
Why
A couple of months back, I got deep into a game called Men of War : Assault Squad 2. It’s a fantastic WW2 RTS, with a focus on realism, large scale battles and a hardcore attention to details. Every single unit/vehicle have their own independent inventory, holding their own ammo and items. Tank battle are interesting as the armor system is pretty detailed and is not simply built around hitpoints.
While the base game itself is pretty good, what makes the game truly unique is the insane amount of mods that you can download to improve/overhaul the experience. Do you want to play as the French army resisting the German invasion? You can! Do you want to take the role of the German army retreating the Stalingrad disaster? It’s there.
Or do you want to play as Star Wars clone troopers or Wargammer 40k Space marines?
It truly elevates a pretty good game into something that you can spend a lot of time with.
So naturally, I got into modding which means downloading the mods with the Steam workshop and that’s where I had a small problem with the interface.
The mods I was downloading we’re pretty old. The game itself was released in 2014 and while the modding scene is still active, it’s slowing down.
Pain
Looking at the mod page for each individual mod, you can only:
- See the last 40 comments.
- No search feature
- No way to upvote helpful/constructive comments.
This makes it very hard to find some specific error messages or see if someone is having the same issue has you.
60 pages of comments to manually go through…
Solution
Let me preface this by saying that I am not a full stack developer. I dabble in web stuff with FastAPI/Python but no where near enough to feel comfortable with the millions of javascript frameworks/ajax. Fancy front end stuff is a black box to me.
Looking at the comment page, I figured I might be able to find a way to extract the comments. Either through parsing the HTML directly or finding a way to talk to the backend directly.
Let’s look at what happens when you first load the workshop mod page –> https://steamcommunity.com/sharedfiles/filedetails/?id=569264526
We can see that the initial HTML returned already contains the comments.
So that is not a huge help as we’d like to get all the comments for that workshop project.
Let’s try to navigate to the dedicated comments page –> https://steamcommunity.com/sharedfiles/filedetails/comments/569264526
It’s a bit better, but we also just see the HTML returned with the comments pre-baked in. That said, we can also see that it seems to be a dynamic process.
So, if we try to change the comments page number, it should refresh the content and we might be able to catch it in the act!
Bingo!
We can now look at the POST itself and find what data is being sent to the backend.
Chrome will let us conver the command to a curl directly.
curl 'https://steamcommunity.com/comment/PublishedFile_Public/render/76561198072547070/569264526/' \ 284ms Sun 05 Dec 2021 09:12:10 PM
--data-raw 'start=50&totalcount=2986&count=50&sessionid=0e5d890eb06926fc29b57e78&extended_data=%7B%22contributors%22%3A%5B%2276561198072547070%22%2C%7B%7D%5D%2C%22appid%22%3A244450%2C%22sharedfile%22%3A%7B%22m_parentsDetails%22%3Anull%2C%22m_parentBundlesDetails%22%3Anull%2C%22m_bundledChildren%22%3A%5B%5D%2C%22m_ownedBundledItems%22%3A%5B%5D%7D%2C%22parent_item_reported%22%3Afalse%7D&feature2=-1' \
--compressed
Executing the curl returns the comments directly!
{"success":true,"name":"PublishedFile_Public_76561198072547070_569264526","start":50,"pagesize":"50","total_count":2986,"upvotes":0,"has_upvoted":0,"comments_html":"\t\t\r\n\t\r\n\t<div class=\"commentthread_comment responsive_body_text \" id=\"comment_3051734364731588414\" style=\"\">\r\n\t\t\t\t<div class=\"commentthread_comment_avatar playerAvatar online\">\r\n\t\t\t\t\t\t<a href=\"https:\/\/steamcommunity.com\/id\/MR_TUKUL\" data-miniprofile=\"191632736\">\r\n\t\t\t\t\t\t\t\t\t<img src=\"https:\/\/cdn.cloudflare.steamstatic.com\/steamcommunity\/public\/images\/avatars\/a7\/a7852b116a0d0a955e3332bbddff133a674af104.jpg\" srcset=\"https:\/\/cdn.cloudflare.steamstatic.com\/steamcommunity\/public\/images\/avatars\/a7\/a7852b116a0d0a955e3332bbddff133a674af104.jpg 1x, https:\/\/cdn.cloudflare.steamstatic.com\/steamcommunity\/public\/images\/avatars\/a7\/a7852b116a0d0a955e3332bbddff133a674af104_medium.jpg 2x\">\t\t\t\t\t\t\t<\/a>\r\n\t\t<\/div>\r\n\t\t<div class=\"commentthread_comment_content\">\r\n\t\t\t<div class=\"commentthread_comment_author\">\r\n\t\t\t\t<a class=\"hoverunderline commentthread_author_link\" href=\"https:\/\/steamcommunity.com\/id\/MR_TUKUL\" data-miniprofile=\"191632736\">\r\n\t\t\t\t\t<bdi>OPPA MANHWA<\/bdi><\/a>\r\n\t\t\t\t\r\n\t\t\t\t\r\n\t\t\t\t\r\n\t\t\t\t\r\n\t\t\t\t\r\n\t\t\t\t\r\n\t\t\t\t<span class=\"commentthread_comment_timestamp\" title=\"30 June, 2021 @ 2:18:56 pm EST\" data-timestamp=\"1625077136\">\r\n\t\t\t\t\t30 Jun @ 2:18pm \r\n\t\t\t\t<\/span>\r\n\t\t\t\t\t\t\t\t<div class=\"commentthread_comment_actions\" >\r\n\t\t\t\t\t\t\t\t\t<\/div>\r\n\t\t\t<\/div>\r\n\t\t\t<div class=\"commentthread_comment_text\" id=\"comment_content_3051734364731588414\">\r\n\t\t\t\tSama Aj Ternyata Mod Ini Juga, Anjing Kontol Memek\t\t\t<\/div>\r\n\t\t\t\t\t\t\t\t<\/div>\r\n\t\t\t<\/div>\r\n\t\t\r\n\t\r\n\t<div class=\"commentthread_comment responsive_body_text \" id=\"comment_3075377162308086469\" style=\"\">\r\n\t\t\t\t<div class=\"commentthread_comment_avatar playerAvatar online\">\r\n\t\t\t\t\t\t<a href=\"https:\/\/steamcommunity.com\/profiles\/76561198852158232\" data-miniprofile=\"891892504\">\r\n\t\t\t\t\t\t\t\t\t<img src=\"https:\/\/cdn.cloudflare.steamstatic.com\/steamcommunity\/public\/images\/avatars\/4c\/4c77b8c939d2e7f3063aa7538072536a2d6ad667.jpg\" srcset=\"https:\/\/cdn.cloudflare.steamstatic.com\/steamcommunity\/public\/images\/avatars\/4c\/4c77b8c939d2e7f3063aa7538072536a2d6ad667.jpg 1x, https:\/\/cdn.cloudflare.steamstatic.com\/steamcommunity\/public\/images\/avatars\/4c\/4c77b8c939d2e7f3063aa7538072536a2d6ad667_medium.jpg 2x\">\t\t\t\t\t\t\t<\/a>\r\n\t\t<\/div>\r\n\t\t<div class=\"commentthread_comment_content\">\r\n\t\t\t<div class=\"commentthread_comment_author\">\r\n\t\t\t\t<a class=\"hoverunderline commentthread_author_link\" href=\"https:\/\/steamcommunity.com\/profiles\/76561198852158232\" data-miniprofile=\"891892504\">\r\n\t\t\t\t\t<bdi>A Random Knighty Boi<\/bdi><\/a>\r\n\t\t\t\t\r\n\t\t\t\t\r\n\t\t\t\t\r\n\t\t\t\t\r\n\t\t\t\t\r\n\t\t\t\t\r\n\t\t\t\t<span class=\"commentthread_comment_timestamp\" title=\"20 June, 2021 @ 8:49:17 am EST\" data-timestamp=\"1624193357\">\r\n\t\t\t\t\t20 Jun @ 8:49am \r\n\t\t\t\t<\/span>\r\n\t\t\t\t\t\t\t\t<div class=\"commentthread_comment_actions\" >\r\n\t\t\t\t\t\t\t\t\t<\/div>\r\n\t\t\t<\/div>\r\n\t\t\t<div class=\"commentthread_comment_text\" id=\"comment_content_3075377162308086469\">\r\n\t\t\t\tWill MW3 be added or compatibility with any cheats mod?\t\t\t<\/div>\r\n\t\t\t\t\t\t\t\t<\/div>\r\n\t\t\t<\/div>\r\n\t\t\r\n\t\r\n\t<div class=\"commentthread_comment responsive_body_text \" id=\"comment_3075377162306191762\" style=\"\">\r\n\t\t\t\t<div class=\"commentthread_comment_avatar playerAvatar offline\">\r\n\t\t\t\t\t\t<a href=\"https:\/\/steamcommunity.com\/profiles\/76561198853416197\" data-miniprofile=\"893150469\">\r\n\t\t\t\t\t\t\t\t\t<img src=\"https:\/\/cdn.cloudflare.steamstatic.com\/steamcommunity\/public\/images\/avatars\/89\/898bf458d57db46ac065dcfdc1484bc0f
We can even pass it to jq directly and pretty-fi the output a little bit.
{
"success": true,
"name": "PublishedFile_Public_76561198072547070_569264526",
"start": 50,
"pagesize": "50",
"total_count": 2986,
"upvotes": 0,
"has_upvoted": 0,
"comments_html": "\t\t\r\n\t\r\n\t<div class=\"commentthread_comment responsive_body_text \" id=\"comment_3051734364731588414\" style=\"\">\r\n\t\t\t\t<div class=\"commentthread_comment_avatar playerAvatar online\">\r\n\t\t\t\t\t\t<a href=\"https://steamcommunity.com/id/MR_TUKUL\" data-miniprofile=\"191632736\">\r\n\t\t\t\t\t\t\t\t\t<img src=\"https://cdn.cloudflare.steamstatic.com/steamcommunity/public/images/avatars/a7/a7852b116a0d0a955e3332bbddff133a674af104.jpg\" srcset=\"https://cdn.cloudflare.steamstatic.com/steamcommunity/public/images/avatars/a7/a7852b116a0d0a955e3332bbddff133a674af104.jpg 1x, https://cdn.cloudflare.steamstatic.com/steamcommunity/public/images/avatars/a7/a7852b116a0d0a955e3332bbddff133a674af104_medium.jpg 2x\">\t\t\t\t\t\t\t</a>\r\n\t\t</div>\r\n\t\t<div class=\"commentthread_comment_content\">\r\n\t\t\t<div class=\"commentthread_comment_author\">\r\n\t\t\t\t<a class=\"hoverunderline commentthread_author_link\" href=\"https://steamcommunity.com/id/MR_TUKUL\" data-miniprofile=\"191632736\">\r\n\t\t\t\t\t<bdi>OPPA MANHWA</bdi></a>\r\n\t\t\t\t\r\n\t\t\t\t\r\n\t\t\t\t\r\n\t\t\t\t\r\n\t\t\t\t\r\n\t\t\t\t\r\n\t\t\t\t<span class=\"commentthread_comment_timestamp\" title=\"30 June, 2021 @ 2:18:56 pm EST\" data-timestamp=\"1625077136\">\r\n\t\t\t\t\t30 Jun @ 2:18pm \r\n\t\t\t\t</span>\r\n\t\t\t\t\t\t\t\t<div class=\"commentthread_comment_actions\" >\r\n\t\t\t\t\t\t\t\t\t</div>\r\n\t\t\t</div>\r\n\t\t\t<div class=\"commentthread_comment_text\" id=\"comment_content_3051734364731588414\">\r\n\t\t\t\tSama Aj Ternyata Mod Ini Juga, Anjing Kontol Memek\t\t\t</div>\r\n\t\t\t\t\t\t\t\t</div>\r\n\t\t\t</div>\r\n
We can reduce some of the parameters.
curl -Ss 'https://steamcommunity.com/comment/PublishedFile_Public/render/76561198024853941/438074008/' --data-raw 'start=0&totalcount=0&count=1' | jq .
{
"success": true,
"name": "PublishedFile_Public_76561198024853941_438074008",
"start": 0,
"pagesize": "1",
"total_count": 1008,
"upvotes": 0,
"has_upvoted": 0,
"comments_html": "\t\t\r\n\t\r\n\t<div class=\"commentthread_comment responsive_body_text \" id=\"comment_3192486000803793562\" style=\"\">\r\n\t\t\t\t<div class=\"commentthread_comment_avatar playerAvatar online\">\r\n\t\t\t\t\t\t<a href=\"https://steamcommunity.com/profiles/76561198449656125\" data-miniprofile=\"489390397\">\r\n\t\t\t\t\t\t\t\t\t<img src=\"https://cdn.akamai.steamstatic.com/steamcommunity/public/images/avatars/14/14be41ce898380c053c44bdc4228e6dcb4b69a6e.jpg\" srcset=\"https://cdn.akamai.steamstatic.com/steamcommunity/public/images/avatars/14/14be41ce898380c053c44bdc4228e6dcb4b69a6e.jpg 1x, https://cdn.akamai.steamstatic.com/steamcommunity/public/images/avatars/14/14be41ce898380c053c44bdc4228e6dcb4b69a6e_medium.jpg 2x\">\t\t\t\t\t\t\t</a>\r\n\t\t</div>\r\n\t\t<div class=\"commentthread_comment_content\">\r\n\t\t\t<div class=\"commentthread_comment_author\">\r\n\t\t\t\t<a class=\"hoverunderline commentthread_author_link\" href=\"https://steamcommunity.com/profiles/76561198449656125\" data-miniprofile=\"489390397\">\r\n\t\t\t\t\t<bdi>egarlock</bdi></a>\r\n\t\t\t\t\r\n\t\t\t\t\r\n\t\t\t\t\r\n\t\t\t\t\r\n\t\t\t\t\r\n\t\t\t\t\r\n\t\t\t\t<span class=\"commentthread_comment_timestamp\" title=\"1 December, 2021 @ 3:37:01 pm PST\" data-timestamp=\"1638401821\">\r\n\t\t\t\t\t1 Dec @ 3:37pm \r\n\t\t\t\t</span>\r\n\t\t\t\t\t\t\t\t<div class=\"commentthread_comment_actions\" >\r\n\t\t\t\t\t\t\t\t\t</div>\r\n\t\t\t</div>\r\n\t\t\t<div class=\"commentthread_comment_text\" id=\"comment_content_3192486000803793562\">\r\n\t\t\t\tmust update\t\t\t</div>\r\n\t\t\t\t\t\t\t\t</div>\r\n\t\t\t</div>\r\n",
"timelastpost": 1638401821
}
Next up, we’ll talk about building a small web frontend to make this a bit more dynamic and query other Workshop projects on demand.