InfoStr regression tests: running them with rtest

Some of the regression tests currently fail if the new rtest.py system is used.

It seems like in all the previous cases, rtest was discovering something that the old system was missing, so, in most cases gold profiles had to simply be updated.

I just wanted to check that this is also the case here.

Gold:

15%20AM

Current, for the same item:

47%20AM

So the old system isn’t looking at ICONS, though it is a bit puzzling to me that there is two MRS in current?

Two MRSes in current suggests that the current grammar (= grammar generated by the current customization system) is finding ambiguity. Can you test that example by hand to confirm?

It’s also suspicious that one of those two has an ICONS list and the other doesn’t.

@olzama yes, I noticed this and mentioned it in the email thread to matrix-dev (the message sent around Aug 6). Here’s the tests that were failing:

  • infostr-foc-affix-after-noun
  • infostr-foc-affix-obj
  • infostr-foc-affix-obj-verb
  • infostr-foc-affix-subj
  • infostr-foc-affix-subj-obj
  • infostr-foc-affix-subj-obj-hier
  • infostr-foc-affix-subj-obj-verb
  • infostr-foc-affix-subj-verb
  • infostr-foc-sov-prev

I think the old comparison function do not examine ICONS.

@ebender The missing ICONS is actually a PyDelphin thing (consider the indented, colored MRSs you’re seeing are not the MRSs directly out of the grammar, but those read and written by PyDelphin, which is also why they show TOP and not LTOP). It was a feature that PyDelphin does not print ICONS if the list were empty, because I didn’t want to start printing ICONS lists for grammars that didn’t implement ICONS at all. Perhaps this feature is now a bug, or at least it needs to be refined to distinguish empty-list from no-list.

Well the issue is that the test passes with the old system and fails with rtest…

Right; but why is there two MRS in the current profile, is that actual ambiguity, you think? I think I did not fully understand your explanation about ICONS.

So I checked the gold profile to confirm there is indeed only one result there for this item (confirmed).

So it looks like:

(1) The old regression test system failed to notice the ambiguity that was introduced at some point.
(2) Either the tests (which Michael lists above) should be updated or the bug should be found and fixed in the customization system.

@sanghoun Can you still tell whether there should or should not be ambiguity of this sort:

The NP->NP rule is a “narrowly focused phrase”, and the top S is a special NF-phrase as well.

Note that the MRS looks the same however there is a difference in ICONS (see the first post) as well as in the shape of the tree.

@olzama yes, the ambiguity is what is is. Even without such differences, differences in ICONS were ignored but no longer with rtest.py.