Grid Sage Forums

Grid Sage Forums

  • November 25, 2024, 01:46:39 AM
  • Welcome, Guest
Please login or register.

Login with username, password and session length
Advanced search  

News:

LINKS: Website | Steam | Wiki

Author Topic: Cogmind Dataminer Scoresheet Viewer  (Read 2172 times)

leiavoia

  • Unaware
  • *
  • Shared a Confirmed Combat Win
  • Posts: 9
    • View Profile
Cogmind Dataminer Scoresheet Viewer
« on: October 27, 2021, 08:26:08 PM »

(Posting this here for posterity and reference. Previously posted on Discord.)

Cogmind Dataminer is an online scoresheet viewer for your public end-of-game scoresheets. It takes raw data from your scoresheet and makes many pages of pretty graphs and charts of your adventure. You can also view scoresheets of any other player's game on public record.

The app:
https://leiavoia.net/cogmind/dataminer/

The code:
https://github.com/leiavoia/cogmind-dataminer

How to find your scoresheets:
https://leiavoia.net/cogmind/dataminer/help.html

Example game (mild spoilers)
https://leiavoia.net/cogmind/dataminer/?osF59nAns4warikxj

The app is written entirely in JavaScript, so it runs directly in your web browser. You can access any scoresheet that is automatically uploaded by Cogmind when your run ends (you need to enable this in your settings). You can also manually upload .json scoresheets if you have those.

This app was written for data available in Beta 10.2 and may be updated as data formats change for newer versions of the game.
Logged

leiavoia

  • Unaware
  • *
  • Shared a Confirmed Combat Win
  • Posts: 9
    • View Profile
Technical Stuff
« Reply #1 on: October 27, 2021, 08:55:10 PM »

@Kyzrati:

Installation

If you want to, you can easily integrate this directly into the scoresheets area of gridsagegames.com. This app is super easy to install. You literally just copy the files onto your server and change the scoresheet links and it works. You don't need to build, compile, configure, or enable anything. I think it would make a nice addition. More info in the README.

Updates

I may update the app when you update the data format on github. I have your cogmind-scoresheet project watched so i will be alerted if you update it.

Suggestions

I have a few suggestions you might work into your next release as time and attention allow:

No JSON file is saved locally on the player's HDD when games are over. This means that 1) players cannot use the Dataminer app offline, and 2) player's that have not elected to upload scores have zero access to JSON structured data and can never use Dataminer for their runs. I'm not sure if outputting.json is easy to do from the cogmind executable, but if it is, that would be a big plus.

Dataminer can be run completely offline by manually "uploading" a JSON scoresheet. The scoresheet is read directly by the browser's javascript, so no server is required. If you can get local JSON output working and elect to ship the game with Dataminer (open source), you can integrate the two very easily with minor adjustments. Players could then view review their games right on their own machine.

The scoresheet hash (the filename used by cogmind-api.gridsagegames.com) is not contained in the file itself. Therefore, the file cannot reference itself using data from within the scoresheet. This creates some problems with locating the file in the first place. It can currently be found at the bottom of the .txt scoresheet output (if uploaded), or in the links on the leaderboards (if uploaded). Ideally, please add `meta.filehash` or similar to .json files.

Scoresheets contain many additional filenames including `header.filename`, `meta.runGuid`, and `meta.runId`. It would be best to standardize these. (This seemingly unimportant detail will be the key to any future database-backed apps in the future). Relational Databases prefer integers for keys, but non-enumerable hashes are better for security if that's important.

The S3 filehash currently being used has no consistent character length. It doesn't NEED to have one, but it would be nice for standardized API usage.

Overall, excellent work on creating all of this wonderful data. Thanks for going the extra mile!  :)


« Last Edit: October 27, 2021, 08:58:20 PM by leiavoia »
Logged

Kyzrati

  • Administrator
  • True Cogmind
  • *****
  • Posts: 4479
    • View Profile
    • Cogmind
Re: Cogmind Dataminer Scoresheet Viewer
« Reply #2 on: October 29, 2021, 01:04:46 AM »

Yay, thanks for all the work on this so far, leiavoia, I've checked it out a bit myself and will be happy to see it get a lot of use over time :D

If you want to, you can easily integrate this directly into the scoresheets area of gridsagegames.com.
It's funny you mention, because I was actually intended to do this, but didn't want to do it unilaterally so asking your permission was on my list of things to do ;)

I have your cogmind-scoresheet project watched so i will be alerted if you update it.
Yeah I decided to only update that one with public releases, rather than private prereleases, so even though I have the fully up-to-date .proto file for the most recent version, it doesn't include any new features/content (such as the latest new map, or anything else that's been added for Beta 11).

No JSON file is saved locally on the player's HDD when games are over. This means that 1) players cannot use the Dataminer app offline, and 2) player's that have not elected to upload scores have zero access to JSON structured data and can never use Dataminer for their runs. I'm not sure if outputting.json is easy to do from the cogmind executable, but if it is, that would be a big plus.
Oh it's certainly easy, in fact I already have it doing that on my own dev version, to be able to inspect the actual JSON data and make sure that has been properly updated each version as well (since that also goes to the server). I'm not sure whether I'd want it on by default for players, just to avoid creating too many files and adding to confusion (fewer people will need the JSON compared to the more accessible txt, until/unless of course they discover Dataminer :P), though having it be an option could totally be a thing. Like right now we already have optional HTML output for the full message log, for example, there was just never any real need for a JSON export in the past.

Dataminer can be run completely offline by manually "uploading" a JSON scoresheet. The scoresheet is read directly by the browser's javascript, so no server is required. If you can get local JSON output working and elect to ship the game with Dataminer (open source), you can integrate the two very easily with minor adjustments. Players could then view review their games right on their own machine.
Wow that sounds like a neat idea, and I do love self-contained projects like that which can be run locally. Will have to keep it in mind (okay, notes :P) as a possibility.

The scoresheet hash (the filename used by cogmind-api.gridsagegames.com) is not contained in the file itself. Therefore, the file cannot reference itself using data from within the scoresheet. This creates some problems with locating the file in the first place. It can currently be found at the bottom of the .txt scoresheet output (if uploaded), or in the links on the leaderboards (if uploaded). Ideally, please add `meta.filehash` or similar to .json files.

Scoresheets contain many additional filenames including `header.filename`, `meta.runGuid`, and `meta.runId`. It would be best to standardize these. (This seemingly unimportant detail will be the key to any future database-backed apps in the future). Relational Databases prefer integers for keys, but non-enumerable hashes are better for security if that's important.
Yeah the hash location thing is something which I've had to work around a bit myself and found it somewhat inconvenient. I wouldn't have done it that way myself but the project was ended before full completion and testing, and isn't something I can update myself, so I don't see that being addressed unless at some point the entire thing is picked back up again for further tweaking.

RE the variables, the filename is just referring to the name of the txt file, thrown in there as a record for convenience (the content includes multiple types of data and it's not needed for JSON purposes, unless you wanted to recreate an alternative version of the scoresheet or something?); as commented in the file "run_id" isn't even a thing anymore, just left over from the old pre-protobuffer system; the only variable that matters on a run-wise basis is its GUID.

The S3 filehash currently being used has no consistent character length. It doesn't NEED to have one, but it would be nice for standardized API usage.
Yeah that surprised me at first, too (and caught me out as well when I first started using it myself!), though I didn't build it so I just went along with it :P
Logged
Josh Ge, Developer - Dev Blog | @GridSageGames | Patreon