No parses when adding regression tests

ace
regression_tests

#1

I’ve added some regression tests without errors and it looks like the gold version has no parses. I’m running out of ideas there to track down what’s going wrong.
If I run the regression tests they succeed and if I load the grammar in regression_tests/grammars into the lkb, everything parses that should. Then if I break something on purpose (eg. commenting out the function call to my library) the tests still succeed and if I look at the logs, I see “0 left results” and “0 right results”. Am I correct that that means that the gold profile has 0 parses? My only guess at this point is that something is wrong in my testsuites, although I can’t see what. (here is an example):
noun1 iverb subord noun2 noun3 tverb aux
*noun1 iverb subord noun2 aux noun3 tverb
*noun1 iverb subord noun2 tverb noun3 aux
*noun1 iverb subord noun2 noun3 aux tverb

Has anyone else run into this?


#2

I think so, yes. You can look at the corresponding gold folder (under /home/gold/) to check.

I’ve added broken regression tests like this several times; not sure what is causing it. My only guess is that it’s copying wrong things to scratch/…


#3

Thanks. I’m pretty sure it’s not a matter of using the wrong files to scratch/. I’ve done that before too, so I’ve been really careful as I’ve been going back and adding everything again. It does look like nothing is parsing in /home/gold/ though so I think whatever is going wrong is happening when the regression test is being added.


#4

Yes, I am very suspicious, too. It seems to me that I carefully check the choices and the testsuite before I add, every time, and I always check the customization system right before… Or at least it seems to me that I do so. And yet… Well I hope we will either train ourselves to be even more careful or will get to the bottom of this, if this is in fact a bug of some kind.


#5

I also wonder how many other tests are like this?.. Perhaps we could add something to the adding script that checks that there are in fact parses in gold… Unless the entire testsuite is ungrammatical.


#6

Adding something to the script that checks is a good idea. I’ve also been wondering if people have added tests with no parses and never noticed. It’s at least good for us to all be aware that we have to double check after adding the test.


#7

I’m going through again and can confirm that I’m copying the right files to scratch. After running the regression tests I can load the grammar in regression_tests/grammars/ into the lkb and parse all of the sentences, but I still have 0 left results and 0 right results in the MRS compare in logs/.

One thing I notice is that all of these regression tests that aren’t “working” (that is, if I break something in my code they still succeed" have “post-reduction lexical gap” on each line in home/gold/mytest/parse. for example:

10@1@10@-1@0@-1@0@0@-1@0@0@-1@0@-1@-1@-1@-1@-1@-1@-1@-1@-1@-1@0@0@0@0@0@-1@-1@-1@-1@34728@-1@-1@-1@23-6-2013 14:28:24@post-reduction lexical gap@

Does anyone know what that means?


#8

Have you tried opening up the gold profile through the [incr tsdb()] podium so you can examine it by hand?


#9

I haven’t been able to- it keeps freezing on me. But I think I’ve found a (somewhat unsatisfying) solution. I just re-wrote one of my testsuites in a fresh file and was able to add it to regression tests. It shows parses now. I’m not sure what was wrong with the old one, but I’m guessing it had a new line or some whitespace that wasn’t visible in the text editor. I’m going to continue doing this for the rest of them and see if I have any other problems.


#10

It seems likely that the profiles are corrupted in some way then. Sounds like recreating them is a good idea.


#11

Corrupted by merging, somehow, perhaps…

It sounds like merging the regression_tests folders should just work, but I
have a strong sense that there is some systematic mess happening. Could be
us not doing the process correctly, but in any case. Perhaps if the tests
themselves change (get updated), and then two copies get merged…


#12

I was able to recreate all of the tests. I think we should all be on the look out when we merge (double check home/gold/results and logs/ in particular) and see if we can get an idea of what’s going wrong when. It’s really hard to pin down how these tests are getting corrupted so long after the fact (I always wonder if I ever added them correctly in the first place)


#13

I am adding a new test right now, and it seems to me that “post-reduction lexical gap” appears there for my ungrammatical things… So perhaps it is normal and the problem is not having this message but rather having it correspond to what should be a grammatical string… Hmm.


#14

My guess is that this has to do with morphological processing. What happens with those items when you parse them interactively with the LKB?


#15

As for the message about “post-reduction lexical gap”, Dan says:

“This error message from ACE occurs when your grammar includes one or more token-mapping (preprocessing) rules which conspire to delete all of the edges from one of the cells in the initial parse chart, so that the parser cannot hope to find a covering analysis. For the ERG, this occasionally happens when I adjust the rules for inserting lexical entries for unknown words, and then try to filter some of those added entries in case I already have a “native” entry for that word. I imagine you must have one or more rules in a file like the ERG’s “lfr.tdl” which discard some unwanted entries before parsing. Try commenting out these rules and see if the error message for that sentence disappears. You can see the error just by trying to parse that one sentence directly with ACE.”

So, looks like this should not be relevant to our regression tests issue.


#16

Unless there’s something amiss with the lexical rules as applied to those examples?


#17

Well, the thing is, I see this same message in tests which produce parses for grammatical sentences just fine. The problematic ones are the ones which end up having 0 parses. So, since I see the message in both types of tests, does not seem like it points to the problem. Could still be the case, but looks more like something else leads to the gold profile having 0 parses in it…


#18

For what it’s worth- my properly added regression tests with ungrammatical examples do NOT have the “post-reduction lexical gap” in the parse file. So I do think it’s pointing to something different and you don’t want it in your tests.


#19

So if “lexical gap” means no possible stem was found in the lexicon, the only reason I can imagine for that on the surface level is grabbing a wrong txt_suite file. But I know, we check every time that it is in fact the right one! Weird.


#20

I was finally able to load the item file into incr tsdb() and process all items. It printed a errors like “No analysis found for token 6-7 ^M” to the emacs window for each sentence. I’m pretty sure that’s the lexical gap. Can you see if this is your issue too? Or since it’s only happening for ungrammatical examples for you, it might have something to do with your * character?