Would it make sense to store gold grammars?

Currently there is no way to load the grammar which was used to produce the gold profile. Would it make sense to start checking those in? Could make debugging regression test failures easier? Or are there good reasons not to do this?

Well, you could get the version of the customization system that corresponds to the creation of the gold profile and create the grammar from it, right?

What kind of svn command would that be? (How do I figure out which commit?) There is of course the previous commit but in some cases (like what we are seeing with rtest), we have lost the info about when the system was last well-behaved with respect to a test.

Give svn log a path and you’ll see commits where that path changed:

$ svn log gmcs/regression_tests/home/gold/infostr-foc-adp-after-noun/
------------------------------------------------------------------------
r40971 | ekayen | 2018-01-25 03:31:53 +0800 (Thu, 25 Jan 2018) | 1 line

merged adnominal possession library into trunk
------------------------------------------------------------------------
r27998 | sanghoun | 2014-02-07 15:16:38 +0800 (Fri, 07 Feb 2014) | 1 line

regression testing with ACE, new profiles with ICONS, and minor debugging in info-str
------------------------------------------------------------------------

I’m not sure what you mean about losing info with rtest or why that’s different from the other regression-testing systems (but I am curious).

Regarding “when the system was last well-behaved”, if you’ve changed regression testing systems and start seeing errors you didn’t before, then it’s not known what commit introduced changes that caused the test to fail. It could have been there from the beginning and the previous system didn’t catch them. You could try a binary search. Say there are 5 commits for a gold profile and the current one is failing. Check out the 3rd commit (in the middle) and test again. If it fails, then go back farther and get the 2nd commit; or if it passes, get the 4th one. After this second test you’ll know which of the 5 commits caused the issue.

That is what I meant; that we switched to rtest and started seeing errors we haven’t seen before. Emily suggested looking at the commit which created the gold profile… Not to find out which change broke it but to look at that gold represents. I still think storing the gold grammars along with the gold profiles could be more convenient though.

Storing gold grammars starting now isn’t going to help with your problem though — and there is a general principle about not storing what can be recreated from what is stored.

The procedure I recommend is as follows:

  1. Find last modified date on the gold profile
  2. Check out the version of the customization system from that date (incl the regression tests directory, which has the choice file)
  3. Recreate the grammar with that system
1 Like

Yes, indeed. Provided locating the right commit is easy (and sounds like it is), I would completely agree with this.

svn log

Well that will return loads of stuff, making the process not easy, but I think Mike was demoing:

svn log filename

Sorry – that’s what I tried to put in, with angle brackets around filename which apparently is mark up for “hide this”…

1 Like