Syntactic parsing for NLP in the LLM era

Publications
1

From a paper I wrote with colleagues about detecting MWE in UD analyses, I got a review:

Although it may be i am not in this specific field, since the BERT model, many downstream NLP tasks does not rely too much on the syntactic parsers. so maybe this may limit the real-world applications of this work;

How to answer that? Any reference about the potential benefits of syntactic analyses despite the LLM promises to deliver everything from the surface forms alone? Does anyone know about any paper that points to any experiment that shows the benefits of having syntactic analyses?

2

I think I would try to argue that there is a ceiling to the “real-world” applications of LLMs and that theoretical understanding of structure continues to be important. In this context parsing is the way to test to what degree our assumptions about syntax correspond to reality. There is probably no point in using LLMs for that (perhaps only to tag sequences which we already understand very well, so, simply automating tasks already solved theoretically) because the whole history of training LLMs is about eschewing theory. They might be able to generate some sequences of e.g. UD tags but what does it give us in terms of better understanding syntax?

1 Like
3

Carlos also suggests this paper: Improving the performance of graph based dependency parsing by guiding bi-affine layer with augmented global and local features - ScienceDirect

1 Like
4

Well, here is one :slight_smile:

Zinda, Eric. The ‘Perplexity’ Natural Language Prototype. InductorSoftware.com, 4 Aug, 2020, The ‘Perplexity’ Natural Language Prototype | Eric Zinda Blog.

If you want to tell a story, you want control over what happens. If you want to build a production natural language front end to software (not a demo), you want determinism and explainability.

Maybe obvious but I wonder if there are other examples that could help on the Applications Page?

5

Thanks, indeed. The paper mentioned by @olzama has many references to recent works in the first paragraph of the introduction advocating the benefits of syntactic analyses for downstream NLP applications.

1 Like
6

I agree, and I believe that almost all DELPH-IN participants also agree with that. What I am looking for is pieces of evidence, experiments, and results that can support this claim. As I said in the summit, people often agree that LLM is insufficient for language study. But for NLP applications, the question is how many tasks (tokenization, sentence segmentation, lemmatization, PoS tag, parsing, NER, WSD, etc) from the traditional NLP pipelines are still relevant.

Let me make it concrete. In the LREC 2016, I attended a workshop about ERG semantics. Examples from the slides advertise the capability of ERG to capture predicate-argument structures among distant dependents and in unusual word orders. By asking chatGPT, we can almost always have good answers. Of course, a careful analysis of the QA below will highlight some potential inconsistencies.

"Most house cats are easy for dogs to chase." who chased whom?

In the given statement, dogs are chasing the house cats.

"Which book did the guy who left give to his neighbor?" who left the book?

The guy who left gave the book to his neighbor.

In "Which book did the guy who left give to his neighbor?", who gave the book to whom?

In the given sentence, the guy who left gave the book to his neighbor.

In "Which book did the guy who left give to his neighbor?", who own the book?

In the given sentence, it is not specified who owns the book.

In "Which book did the guy who left give to his neighbor?", what is the book about?

In the given sentence, the specific book or its content is not mentioned.

In "Which book did the guy who left give to his neighbor?", who was the owner of the book?

In the given sentence, the owner of the book is the guy who left.

In "Which book did the guy who left give to his neighbor?", who owned the book?

In the given sentence, it is not specified who owned the book.

In "We know that the cat didn’t sleep." who knows what?

In the given statement, it is not specified who knows that the cat didn't sleep.

In "We know that the cat didn’t sleep." the person who wrote the statement knows what? 

In the given statement, the person who wrote the statement knows that the cat didn't sleep.

But the predicate-argument structure is only part of the semantics… Sure, I don’t buy the LLM-centered NLP. I am just trying to find references to support that: generative language models are just one tool but do not replace all the others.

I hope this discussion is not too dull.

7

ChatGPT and LLMs are not an oracle capable of answering any question.

image
image3094×2328 657 KB

8

Sorry, I wasn’t trying to be pedantic! I was saying why I needed this tech for my purposes, and why it is an example of what you are looking for (I think).

9

I think the comment is suggesting that detecting MWE is not a real world application or similarly parsing is not a real world application. Yes, if the end goal is to build a search engine (Google) or a dialogue system (Alexa) or a recommendation system (ads), then doing parsing is basically off the table. However, parsing has been largely off the table for these tasks for a long time, so to suggest that this is because of BERT seems ill conceived, either missing new info or missing old info.

There is value in building a syntactic analysis (or MWE recognizer) beyond improving search results.