Tutorial #10: StrictJSON v2 (StrictText): Handle any output - quotation marks or backslash!
11After using StrictJSON myself for a while, I noticed it frequently breaks when it handles codes, or fields which contain quotation marks or backslashes, or unmatched curly brackets, as it interferes with parsing in json.loads()
With StrictJSON v2 (StrictText), we return everything as a string and this does not break at all, based on my extensive testing!
Granted, we will need to post-process the string ourselves, but that is much more flexible because now we can ensure our own datatype match and so on.
The output of StrictJSON v2 is still in a dictionary form, so it will be very easy to parse through the various output fields to process your data.
Try it out, and let me know!
If enough people want it, I can also implement this post-processing part native in StrictJSON v2. Let me know in the comments.
~~~~~~~~~~~~~~
Why use it?
- When you want GPT to output into various components and do not want to parse it yourself (we use JSON because GPT processes it well)
- When you want certainty that all the output fields are present (we have a feedback loop to ensure GPT outputs the relevant fields)
~~~~~~~~~~~~~~
Under the Hood:
- We transform each output format key into {delimiter}{key}{delimiter}
- Default delimiter is '###', you can change it to whatever is not present in your dataset
- We use regex to split on the key with delimiter to extract the key and values and output as a dictionary
- If any of the keys is not in the output, we ask GPT to process again (iterative feedback loop)
~~~~~~~~~~~~~~
Jupyter Notebook: Jupyter Notebook here: https://github.com/tanchongmin/strictjson/blob/main/Strict_Text_(Strict_JSON_v2).ipynb
Github: https://github.com/tanchongmin/strictjson
StrictJSON v1: https://www.youtube.com/watch?v=A6sIh-lmApk
~~~~~~~~~~~~~~
0:00 Introduction and Motivation
4:35 Under the Hood
~~~~~~~~~~~~~~
AI and ML enthusiast. Likes to think about the essences behind breakthroughs of AI and explain it in a simple and relatable way. Also, I am an avid game creator.
Discord: https://discord.gg/bzp87AHJy5
LinkedIn: https://www.linkedin.com/in/chong-min-tan-94652288/
Online AI blog: https://delvingintotech.wordpress.com/
Twitter: https://twitter.com/johntanchongmin
Try out my games here: https://simmer.io/@chongmin