Ceci est une ancienne révision du document !
Addition to last month After last month’s article came out, I got an email from Ian, who suggested his preferred tool for Markdown to HTML conversion, Remarkable (see the Further Reading section for a link). So, for any readers who are looking for something more like that, you now have a starting place! Additionally, I created/updated my bash script for pandoc (md2pdf), which looks like that shown above. If you want to use this script yourself, make sure the path to the tufte.css file is correct for your system (see last month’s article for more details). The script itself is executed like this: md2pdf Notes-To-Convert.md The script automatically creates the PDF filename using the original markdown filename, and adds a title to the PDF to avoid the warning/error appearing. You could also easily expand this script to work for any number of arguments by looping over “$@” and placing the command within the for loop for each item. I typically won’t need this, or, if I do, I’ll likely want to create a single PDF, which would require a different pandoc command anyways, which is why I left the script a little more basic (and easier to understand). And now back to your regularly scheduled programming…
Complément au mois précédent
Après la sortie de l'article du mois dernier, j'ai reçu un mail de Ian, qui suggérait son outil préféré pour la conversion du Markdown en HTML, Remarkable (voir le lien dans la section Pour aller plus loin). Aussi, pour tous les lecteurs qui cherchent quelque chose plus dans ce genre, vous pouvez maintenant disposer d'une base de départ !
De plus, j'ai créé/mis à jour mon script bash pour pandoc (md2pdf), qui resemble à ce qui est montré ci-dessus.
Si vous voulez utiliser vous-même ce script, assurez-vous que le chemin vers le fichier tufte-css est correct pour votre système (voyez l'article du mois dernier pour plus de détails). Le script lui-même s'exécute ainsi :
md2pdf Notes-To-Convert.md
Le script crée automatiquement le nom du fichier PDF en utilisant celui du fichier markdown d'origine et ajoute un titre au pdf pour éviter qu'un avertissement/une erreur se produise. Vous pouvez aussi étendre facilement ce script pour qu'il fonctionne avec une quantité quelconque d'arguments en faisant une boucle au-dessus de « $@ » et en plaçant la commande dans la boucle for de chaque élément. Je n'en ai pas eu vraiment besoin, et c'est le cas, j'aimerais aussi créer un PDF unique, ce qui nécessiterait de toute manière une commande pandoc différente ; c'est pourquoi j'ai conservé le script un peu plus simple (et plus facile à comprendre).
Et maintenant, retournons à notre programmation prévue normalement …
This month, one of the items on my to-do list was to organize my SGF (go game records) files into a format where I can, at a glance, see whether I won or lost, and when I played it. Originally, I had hoped that each file would have the date stored in the SGF information, but it turns out every server has one (or two) variations on data stored in the files. Some had dates, some had copyright information over multiple lines, some had all the metadata on one line, others spread over multiple lines. As such, I decided to make a python script to print out human readable information, without relying on any SGF plugins (as I don’t want the actual moves, just the metadata). Do note, I am condensing the entire process for the sake of this article. My goal is to instill the TDD mindset on my readers, while offering some examples. The full code will be linked at the end of the article, for anyone who wants to pick it apart.
First Step The first step was to decide which format to start with - I settled on the Fox Go Server format, as the information was on one line, and should therefore be the least amount of processing to get the information into Python.
Second Step Once I had decided what to tackle first, I then set up my folder structure like this: sgf.py init.py _tests.py main.py The main.py file I originally added after finishing the SGF class and the tests, but it won’t hurt anything to have the file ready from the beginning. Also, init.py is empty, but seems to be required for relative imports to work.
Third Step - Tests Now for the first file - tests. Following the practices of TDD (and Adam Wathan’s method), I started with my tests instead of any actual code. The starting _tests.py file looked like this: import unittest from sgf import SGF class SGFItemTests(unittest.TestCase): sgfPath = “./fgs-test.sgf” def test_load_singleLine_sgf(self): testItem = SGF(self.sgfPath) self.assertEqual(testItem.getTitle(), “2019-03-03 - Black (16 kyu) VS White (16 kyu) - B+20.50”) if name == 'main': unittest.main() I left it at that, knowing the test would fail. I was also getting warnings and errors from Visual Studio Code about the class not existing before running anything. As such, I skipped running the test and instead worked using the warnings from Code. If, however, this is your first TDD project, I recommend getting in the habit of running the tests at every stage and dealing with the errors as they appear.
Fourth Step - Actual Development sgf.py import re #this is required for the regex code in the future class SGF: def init(self, path): self.title = “created” All I did here was make sure I could import the python file and that it had a constructor. I then began running the tests, and fixing each error as it occurred. First it required me to create a getTitle() function, then I expanded the constructor to loop through the file path and pass each line through to a createTitle function that checked for the existence of specific data (such as PB[], PW[], date[], WR[], BR[], and RE[]). Those fields are player (black), player (white), the date, the players’ ranks, and the result.
Admittedly, I stretched those steps out slowly - first I tried to grab the player names and had my test written for that, and so on, evolving both the class and my tests. For the sake of this article, I’m condensing some steps. The regex I used was as follows: name = re.search('PW\[(.+?)\]', string) if name: white['name'] = name.group(1)
The important part of this code are the normal brackets “()”, which creates a group of all the characters between the square brackets (which are the values I’m after). The name.group(1) line simply loads the saved group into a string. I changed the value I was looking for, but the basic framework remained the same. As you can see, I started saving dictionaries for the various values to make the code more readable. Essentially the entire class became a series of functions to strip out corresponding information (player information, results, date), and the information was then fed back into a class-wide dictionary called “info”. The getTitle function eventually became a function that simply reads the information out of info, and parses it into a nice string. I also expanded my tests to check various sub items (instead of just the title) by creating a function called getPlayers, and then checking the various fields.
Fifth Step - Next Test The entire above step was dedicated to having my test “test_load_singleLine_sgf” pass successfully. The reason I did it this way was as a proof of concept, and to refine the various functions for parsing the data. This means that all I had left to do was upgrade my file parsing function to not fail when all the metadata isn’t on one line. It doesn’t matter if there are extra items, as the regex will pick out only what I’m looking for. I then created a new test called “test_load_multiLine_sgf”, and let it load a game from OGS, which was split up over multiple lines. The first goal was to again load the player data properly (both black and white), which required me to devise a check for whether or not the metadata was over multiple lines. I opened up an online regex tester, put in some test data, and experimented a bit until I found a regex that seemed to work.
The entire checkMultiline function ended up looking like this: def checkMultiline(self, string): multiline = re.search('[a-zA-Z]+\[.+?\]\n', string) if multiline: return True else: return False What the regex does is to search for any characters (upper or lowercase) that precede a square bracket, some characters, a closing square bracket, and a newline. I wasn’t too worried about only matching exactly the metadata lines, as I never read the entire file (I break out of the loop once I find all the information I need), and the secondary regex will not be affected. The check is used in my readSGF function, and every line that matches the multiline check is then strung together into a single string (without newlines), which is passed through to the various functions.
This worked fine for OGS (except reviews) files, and then I tested it on Pandanet (IGS) files, where it promptly broke. The reason it broke was simple - Pandanet added a Copyright value into the metadata, and spread it over 4 or 5 lines (depending on where the SGF was created). I put Pandanet in a separate test, and focused only on that test. Running a single test in Python is as simple as: python _test.py SGFItemTests.test_load_pandanet_sgf. I quickly concluded that using regex for this particular case was going to be tricky, as the number of lines wasn’t always uniform. Instead, I decided to adapt my readSGF function to simply not process the following lines when it discovers the Copyright value.
I do this by initializing a tempCount at 0, and setting it to a value of 6 when I can find “CoPyright[\n” in the string. I also added an ‘if’ to see if tempCount is greater than 0, and when it is, the counter is reduced by one and the loop follows the “continue” directive (where it jumps to the next item in the loop). This effectively skips the plain english lines of text, removing the problems. I also noticed that some SGF files had a CP[] copyright line (such as the OGS review files), which was shorter than CoPyright. As such, I simply initialized tempCount at 5, instead of 6, which worked fine. The only reason I could do this was that the copyright notices always appeared before the game information, which means I didn’t need to take that into consideration. I realize that this last section can be confusing to read. However, this is pretty much the final file, so viewing the links below should help clarify things. There were a few steps afterwards (such as when a file had no date), but they were simple enough to catch and solve when listening to the tests and batch running the file.
Conclusion Anyone who follows the link to the Gist will notice a few things. Firstly, I sanitized the test files to remove any identifiable information. Especially since readers won’t have my test files and will therefore need to adjust the tests, I felt it helpful to label the information more generically. Secondly, there’s a bash file included. The reason for this is simple - I didn’t want to install the python script into a folder in my $PATH, as it would include other files as well and break the tests. Instead, I wrote the bash script in my $PATH, which appends the full path to the files, and then runs the Python script within its actual folder with absolute paths. You’ll need to adjust the path to main.py for your own system. I hope this look into my TDD process might help inspire some readers to give it a shot, just as I have been inspired by others. Also, if there are any fellow Go players out there - perhaps you’ll find this tool useful for organizing your own SGF files. If you have any questions, suggestions, or comments, they can be directed to me at lswest34+fcm@gmail.com.
Homework (Optional) My own goal for this script it to expand it over time. My first revision would be to add a stats calculation system, which will give me the overall stats across all the servers I play on (perhaps even details on wins against stronger/weaker opponents). If any reader is so inclined, feel free to take this suggestion and use it as practice yourself!
Further Reading http://remarkableapp.github.io/ - Remarkable App’s website https://gist.github.com/lswest/1e7fe8751e0d77f880db7d0a266e652f - A Gist containing all my code for this article.