[incr tsdb()] quitting after hitting some kind of error

Uncategorized
1

Using the Ubuntu+LKB image that @bmgraves created for this quarter’s class, we’re running into a new (to me) error when trying to process test suite profiles with [ incr tsdb () ]. Specifically, it seems like Process | All items will give up after … something … happens.

For the specific corpus I’m working with (Meitei, mni), if I try Process | All items, it shows the following in the common lisp buffer in emacs and doesn’t record any parses:

close-connection(): anonymous/corpus/22-01-10/lkb' expiry. main(): no history file .tsdb_history’.
retrieve(): found 1785 items (0 output specifications).
create-cache(): write-through mode for anonymous/corpus/22-01-10/lkb'. largest-run-id(): largest run-id’ is 0.
largest-parse-id(): largest parse-id' (for run’ 1) is 0.
(0) khútlə́m oy-nə , yam waŋ-nə hay-khət-nə kəri=no təw-pi-tə-lə́gə-čhú , tə́rahum=tə oy-pə-čhú parti , koŋgrés ŋə́čhi=təgi čə́hí mə́ŋa-lom=ki mə́-maŋ=tə phə́m-khi-pə , mə́-tú=ki mə́-thə́k čə́hí tə́ramúk=ki mə́-maŋ=tə phə́m-khi-pə , kəya ə́-mə pí-pə ya-í .' [100000] flush-cache(): flushing anonymous/corpus/22-01-10/lkb’ cache … done.
close-connection(): `anonymous/corpus/22-01-10/lkb’ expiry.

If I set the TSQL condition, I can get it to parse a couple, but far from the full test suite. With the TSQL condition set to i-length <= 4, I get:

main(): no history file .tsdb_history'. retrieve(): found 608 items (0 output specifications). create-cache(): write-through mode for anonymous/corpus/22-01-10/lkb’.
largest-run-id(): largest run-id' is 0. largest-parse-id(): largest parse-id’ (for run' 1) is 0. (20) phəǰə-nə kúp-čhin-lə́gə thəm-ləm-í .’ [100000] — error: No analysis found corresponding to token 0-1 PHƏǰƏ-NƏ No analysis found corresponding to token 1-2 KÚP-ČHIN-LƏ́GƏ.
(110) phə-lə-e .' [100000] --- error: No analysis found corresponding to token 0-1 PHƏ-LƏ-E. (150) nípá ŋák=tə́ ləy-u .’’ [100000]
flush-cache(): flushing anonymous/corpus/22-01-10/lkb' cache ... done. close-connection(): anonymous/corpus/22-01-10/lkb’ expiry.

Similarly, if I set it to i-id > 100 I can get an item parsed before it quits. So, I think there is some kind of error that is leading it to give up entirely, but I can’t see what it is. Does anyone have any ideas?

2

@ebender Oh dear, my testing of [incr tsdb ()] hasn’t been up to scratch. I’ve managed to reproduce this issue with another grammar, and have also found that it dates back to the April 2021 LKB-FOS release. The error happens inside [incr tsdb ()], but unfortunately most code there suppresses error signalling, which makes finding the problem more difficult than it should be.

Anyway, below is a temporary fix. For convenience you can put it in your ~/.lkbrc file - but please remember to remove it when you install the next LKB-FOS release.

John

(in-package :lkb)

(defun release-temporary-storage (&key (task :parse))
  (purge-edge-registry)
  (invalidate-marks)
  (invalidate-visit-marks)
  #+:pooling
  (reset-pools :compressp t)
  (case task
    (:parse
     (loop for chart in (cons *chart* (if *active-parsing-p* (list *achart*)))
         do
         (loop
             for i from 0 below (array-total-size chart)
             do
             (loop for cc in (row-major-aref chart i)
                 for edge = (chart-configuration-edge cc)
                 do
                 (loop
                     for tdfs in (list (edge-odag edge) (edge-dag edge))
                     for dag = (and (tdfs-p tdfs) (tdfs-indef tdfs))
                     when (and dag (not (dag-safe-p dag))) do (compress-dag dag)))))
     (clear-chart)
     (setf *parse-times* nil))
    (:generate
     (loop
         for item in %generator-lexical-items%
         for tdfs = (mrs::found-lex-inst-fs item)
         when (tdfs-p tdfs) do (compress-dag (tdfs-indef tdfs)))
     (clear-gen-chart)))
  (loop
      for rule in (append (get-matching-lex-rules nil) (get-matching-rules nil nil))
      for tdfs = (if (or (and (eq task :parse) *chart-packing-p*)
                         (and (eq task :generate) *gen-packing-p*))
                     (rule-rtdfs rule)
                     (rule-full-fs rule))
      for dag = (tdfs-indef tdfs)
      do (compress-dag dag))
  (loop
      with start =
           (case task
             (:parse *start-symbol*)
             (:generate *gen-start-symbol*))
      for root in (if (listp start) start (list start))
      for tdfs = (get-tdfs-given-id root)
      for dag = (and tdfs (tdfs-indef tdfs))
      when (dag-p dag) do (compress-dag dag))
  #+(and :debug :allegro)
  (when (eq task :generate)
    (let* ((*terminal-io* excl:*initial-terminal-io*)
           (*standard-output* *terminal-io*))
      (sys:gsgc-parameters)
      (excl:gc)
      (sys:gsgc-parameters)
      (room t))))
3

Thank you, @johnca !