For some reason, some MSI installers are not using fonts that they should, but instead they use some random font for no reason. Installing Windows 98 SE Unofficial Service Pack 3 just caused even more mess with additional errors, which were fortunately fixed after few more restarts.
For Windows 98 SE Unofficial SP3 issue: Simply wait for about a minute before rebooting and then do it, it's most likely not done yet even if installation says it is.
The solution to MSI installer issue?
Step 1: Find the font that is being used and delete it.
Step 2: If you don't like the next font that's being used, repeat step 1. If you have no more fonts to delete, go to step 3.
Step 3: Regret doing this for a specific MSI installer layout.
In my case, font substitute feature in Windows did not work, at all, so that's out of question.
25