I’ve been thinking it would be helpful for several projects (matrix, pydelphin, getting more people to use delphin stuff) to package and publish ACE in apt (or similar) and homebrew. I looked into the process and it seems relatively straight forward, though of course it would probably increase the maintenance burden to some degree.
Is there any reason why we haven’t done this or don’t want to do this? Any objections? @sweaglesw?
So, this would be an alternative to just downloading the binary and adding its path to PATH?
I suppose there could be some benefit to that, for people who don’t know what PATH is (me, a few years ago), although I must add that, in my experience, homebrew installations are almost always broken There is always something there that ends up out of date, the error messages are uninterpretable, and then the user (me, a few years ago) gives up…
Are there more benefits I am not thinking of? That’s an interesting topic.
@olzama I’m surprised to hear that experience about homebrew! I’m a big fan and haven’t ever had a problem that homebrew couldn’t fix on its own. I also use homebrew cask a lot, which is an alternative to the App Store for downloading GUI apps that works great.
Anyway, why do this? I think the main reason is that it would enable true install scripts, which also means allowing other projects to get into package managers. If you look at the set up instructions for the matrix or PyDelphin, you’ll see that it’s basically “download these packages, set these environment variables, and download ACE.” I’ve looked into making true install scripts before, but the install ACE thing is always a sticking point. If we used a package manager, it would be a simple conditional/try statement, but without we would need to basically roll our own package manager for each supported OS (which I think for ACE is like Ubuntu, Debian, and macOS?), which is to say, download the latest supported version and dependencies to the right spot and run any scripts.
ACE as a package would also make grammar as package much easier.
I have no objection to ACE as a package. Sounds great. I’ve never looked into how to do it, not do I have the time or inclination to add that to my plate now, but if someone else wants to, I’m all in favor.
Are you running OSX? I think overall, setup tools of this sort tend to be worse in the OSX land than they are in open source linux…
Yeah, I basically only use macOS. I use Ubuntu for DELPH-IN and running tensorflow on the GPU :-p
Hence the questions about making things nicer for macOS users.
Long ago, Eric Nichols maintained an apt repository when he was a student at NAIST called ubuntu-nlp, and it packaged PET, the LKB, Jacy, [incr tsdb()], etc… It was nice but took some effort to maintain. If someone similarly wanted to make one like that but with ACE-based tools, it makes sense to include some
python3-pydelphin package (why the
python3- prefix? See here), but I don’t want to maintain it.
In fact I think an apt-installable package for PyDelphin makes the most sense for those just using the
delphin command-line utility and not the Python library (e.g., as a dependency of some research project). As a dependency you’d want to use a virtual environment and pin the version for reproducibility. An
apt install does not make that easy because it installs to the system libraries which may interact with other libraries on the system. But I’m not completely against the idea because I think it’s possible to do both: a system-wide install and a virtual-environment install for a particular project.
Besides, I think
$ pip install pydelphin
Is just as easy as
$ apt install python3-pydelphin
although I suppose you could make the latter a dependency on some ACE package so they get installed together.
CC-ing Eric in case he wants to comment.
I think it would be great to have these.
I would think the most useful would be:
ace (just the binary)
ace-tools (art, mkprof, fftb and all)
ace-erg (the binaries for the grammar)
python3-pydelphin (to get dependencies)
If we wanted to be super helpful then maybe
lkb-fos (with the binaries and elisp)
delphin-viz (the demo system)
I volunteer to test, but don’t think I can commit to maintaining.
I looked into adding ACE to homebrew today a bit. I made a little progress but it’s not done. @sweaglesw (or whoever), homebrew recommends (but in our case doesn’t require) building from source, which I was looking into doing. However, it looks like repp is only available as a compiled binary? Is the repp source available, too?
@goodmami thanks for the notes! I was thinking the same thing about the delphin command. Interestingly, homebrew explicitly discourages you from making packages of things that are installable via pip, but I agree that the killer feature would be a single command to download pydelphin and ACE in one go, in which case I think that would override the no-pip rule. Of course, for homebrew, since we’d be hosting this in our own repo, it seems we can basically do whatever we want anyway…
@bond, sounds good. Once one is going, it may be easy to get them all going, in which case I can try to do so! I definitely think grammar packages is high priority, so if I can get ace working, then I will probably try the ERG and then pydelphin. Others will be maybe from me. I’m also interested in getting the matrix set up this way, but I think that’s dependent on a bit more refactoring of the matrix first…
@goodmami (or whoever), homebrew prefers to pull from a GitHub account. Once I get the proof of concept worked out, can you help me transfer the homebrew repo to the DELPH-IN GitHub account?
I don’t know if Woodley has SVN or something setup, but you can get and compile the source like this:
$ sudo apt install libboost-regex-dev
$ wget http://sweaglesw.com/linguistics/repp-0.2.2.tar.gz
$ tar xf repp-0.2.2.tar.gz
$ pushd repp-0.2.2/
$ ./configure && make all
Of course there’s…
sudo apt install sweaglesw-ace && pip install pydelphin
Maybe that’s not as fun . Btw I put “sweaglesw-ace” because there’s already an ace package in Ubuntu’s repositories. Not sure if adding our deb repository to
/etc/apt/sources.lst would shadow the other package.
Sure, I can setup a repo for you, but I don’t use macOS so I cannot test. Let me know what you need.
Thanks @goodmami - it looks like I made an incorrect assumption that that tarball was compiled and not source. Sorry! I will check it out!
In another direction, I will put some effort to finish the adjustments in my docker file:
If it, we can already have the full logon distribution, ace, fftb and pydelphin. The docker started a separated environment but the home of the user inside the docker is a directory of your choice in the host machine. This makes easy to work with the tools.
I have recently solved some issues in the compilation of ACE (with the help of @sweaglesw) that I want to add in the dockerfile.
It would be nice to move this repo to the https://github.com/delph-in if people agree with that.
@arademaker I like the idea of a Docker container, but I wouldn’t want the full LOGON distribution in it. If you can just get the LKB and [incr tsdb()] (along with the ACE tools, maybe PET if anyone still uses it) that would be fine, but all 4.6GB would make the container huge with little gain.
When I setup a GitHub action to compile Jacy with ACE on every push, I just used their regular Ubuntu image, instead of a custom one, then downloaded the ACE binary in the running instance. It seems inefficient to hit Woodley’s server and download for each run (it can be cached), but the ACE binary is much smaller than the custom Docker image would be, and the standard image is surely cached and optimized to load fast on GitHub’s runners.
The newly released version of LKB-FOS includes [incr tsdb()], so it might be a good candidate for the Docker container. LKB-FOS is now even more compact than before: 76MB for all binaries and supporting data.
Thanks, John. I think it might be even smaller if we take out the macOS binaries? I don’t think we’d need them for a Docker container.
Good point; that would reduce it to 44MB. Training maxent models would require adding the sdsu
evaluate binaries from LOGON, but I’m not sure whether anyone would want to use [incr tsdb()] for training nowadays.
I now understand why no one had done this yet
I have now shipped Homebrew formula for repp, ACE, and PyDelphin (with ACE+repp). If you’re running macOS and want to try it out, you can do so via this command:
brew install dantiston/delphin/ace, etc. Currently, that command will download and compile everything, but it’s actually pretty fast, so no worries there. There’s a pretty standard method for building binaries and sharing them via
bintray.com, but I haven’t gotten around to it yet.
They’re currently hosted in github.com/dantiston/homebrew-delphin. @goodmami it would be nice if you could help me transfer the homebrew-delphin repo to the delphin user(/organization? idk). Once moving, the command would change to
I’ve looked into setting them up for apt, and given what I learned from homebrew, it shouldn’t be a huge amount of work, but I don’t have an ubuntu set up right now, so I probably won’t be able to get to it for a bit more time. It may also be easier to do once
bintray is set up for homebrew…
P.S. I didn’t name the homebrew pydelphin py3pydelphin (though I seriously considered it) because Homebrew seems to discourage python2 dependencies and only one formula distinguishes.
I could not transfer since I don’t own the repository, but I created a new repository with the same name and description and invited you as admin. Then you push your local repo to this new remote:
git remote add origin https://github.com/delph-in/homebrew-delphin.git
git push -u origin master
This should be the same as a transfer except it doesn’t do the following
- copy issues, pull requests, wiki, etc. (you don’t have any of these anyway)
- setup a redirect from dantiston/homebrew-delphin
- remove dantiston/homebrew-delphin; you can delete it yourself
python3- prefix is just for the Debian packaging guidelines (i.e.,
apt packages). I don’t know if Homebrew has any of its own conventions.
I suggest more tests before adding the info in the http://moin.delph-in.net/AceInstall?action=diff&rev1=5&rev2=6
% brew info delphin/delphin/ace
Error: No available formula with the name "delphin/delphin/ace"
Please tap it and then try again: brew tap delphin/delphin
==> Tapping delphin/delphin
Cloning into '/usr/local/Homebrew/Library/Taps/delphin/homebrew-delphin'...
remote: Repository not found.
fatal: repository 'https://github.com/delphin/homebrew-delphin/' not found
Error: Failure while executing; `git clone https://github.com/delphin/homebrew-delphin /usr/local/Homebrew/Library/Taps/delphin/homebrew-delphin` exited with 128.