VNDB is now available on Docker

Posted in

#1 by roggia
2017-07-19 at 05:41
NOTE: This thread is mainly for developers and uses technical language.
NOTE: The docker image is still in alpha version and will be available in the docker hub once it will be finished.

I recently created a docker image for VNDB in order to make things easier for developers and maintainers. This will setup your environment within a few minutes and you won't have to worry about dependencies, flow and database initialization.

The following are the current features of this image:
- Full setup of the VNDB code
- Easy setup with a single command (docker-compose up)
- Database is included
- Live code support

Preview:
link
link
link

Version info:
- apache 2.4
- postgresql 9.6
- imagemagick 7.0.6
- libjpeg 9.0b
- libpng 1.6

Known Issues:
- Live code is enabled only for the "lib" directory (but it should be enough for most of the operations)
- SQL queries and updates are tedious (but you can make them yourself by accessing the database directly with a client)
- Some functionalities of the Makefile are not supported (multi-stop, multi-start, multi-restart)
- The current flow of GIT is messy and will be fixed soon

If you are interested in how I build the image please take a look at this file:
link

Here are the vendor list of all the code dependencies, if you add a new dependency in the list it will be automatically be downloaded during the build process:
link

This is the pre-configured docker config.pl:
link

HOWTO:
- Download docker for your platform at link
- (Optional - Only Windows and Mac users) Install Kitematic at link
- Clone my git repository at link (this will become link after the merge)
- (Only Windows users) Make sure that the ending lines are correct: link
- Execute "docker-compose up" in the cloned repository
- Visit link

That's it, you are ready to go!
#2 by deepweb666
2017-07-20 at 09:58
What is the purpose of this?
#3 by roggia
2017-07-20 at 18:43
Developers can now setup the development environment in a few minutes without having to worry about the installation and configuration.
This will (I hope) make it easier for developers to contribute to the website.
#4 by tigershark
2017-12-06 at 13:11
Any reason not to bind the data folder too and make a .bat/.sh to exec "make all" on the container?
#5 by yorhel
2017-12-09 at 09:05
I've now finally decided to give docker a try, too. I started from scratch as a learning experience, and deviated from roggia's approach:

- Somewhat opposed to the docker philosophy, I put everything inside a single container.
- The docker image uses Ubuntu packages as much as possible, avoiding manual compilation.
- Uses TUWF's new internal web server instead of Apache. This simplifies config a bit and makes for easier debugging.
- The Postgres DB is stored in a named volume.
- The entire VNDB git repo is mounted from the host to the container, so you can work outside of the container while the code runs inside of it.

Usage instructions are on the VNDB git in the README. I've only tested this on Linux, as I haven't the slightest clue how docker works on Windows/Mac. I plan to write a little script to automatically make-and-reload when a file has changed to further simplify this workflow.

EDIT: Done, this script now automatically runs in the docker image. It also works great in non-docker setups.

(And I don't know why, but docker is ridiculously slow on my system...)Last modified on 2017-12-09 at 12:55
#6 by tigershark
2018-01-04 at 19:18
For some reason the db initialization fails for me with yorhel docker. (Scripts don't report any errors but postgres refuses to connect with the given passwords) (and I think changes to the db are lost after a reboot too)

Tried several times and different approaches but had to manually run the commands from the README to get the db up.
#7 by yorhel
2018-01-04 at 19:38
Scripts don't report any errors but postgres refuses to connect with the given passwords
Are you using the recent example config? That one has the vndb_site user/pass that ought to be configured by the setup scripts.

and I think changes to the db are lost after a reboot too
The DB is stored in the 'vndb-data' volume mounted at /var/lib/postgresql. Is your Docker clearing that volume every reboot or is the mount not working?
#8 by tigershark
2018-01-05 at 01:22
Ok, took me the damn whole day to troubleshoot, seems the problem was..... running from an NTFS file system in linux... I guess that messed with permissions and the users you create, manifesting a lot of weird behaviors -.-

Building the docker from a folder in /home works fine, although once up, the web server seems to hang after 20 seconds or so, is there any log I can check?Last modified on 2018-01-05 at 02:29
#9 by yorhel
2018-01-05 at 07:31
Ugh, if Docker for Windows isn't properly emulating a POSIX system, then it's definitely the wrong tool for the job. Unless someone familiar with Docker can help out, I think it's better to look for another tool. I'll play around with Vagrant when I'm bored enough.
#10 by tigershark
2018-01-05 at 14:45
My guess is that using ntfs from windows will work, but I was mixing ntfs and a linux system (cuz docker refuses to start at all on windows with my new hardware, either I'm failing to activate virtualization on bios or the last win10 security update broke something)
#11 by tigershark
2018-01-11 at 14:42
Further testing shows the web server in yorhel's docker hangs if u try to use 2 browsers to access the website.
Login state seems irrelevant, exact page of the site too. Whenever a request is sent from a different browser than the 1st one used (since server boot) will make it hang without ever giving a response to such request.
Since the apache version worked, seems it has to do with some config/bug on the TUWF internal webserver.Last modified on 2018-01-11 at 14:43
#12 by yorhel
2018-01-12 at 09:23
I can't reproduce that issue here, but I just pushed a likely fix. After some playing around I noticed that the vndb-dev-server.pl script did not properly close the connection after the response was sent. The internal web server is just a simple single-process HTTP/1.0 server, so it can only handle one connection at a time. If that connection is not immediately closed, the server will appear to be stuck.
#13 by tigershark
2018-01-13 at 10:44
Works nicely now :)
#14 by alto
2018-10-28 at 18:08
This was pretty painless to get running on docker, especially with the development database option so thanks for that.

The only issue I do have is the search doesn't seem to work for VNs I add/edit. Poking around I think problem is the c_search column not being populated, its blank for every VN I add. If I'm reading it right, I think this should get called on VN change? I'm not very familiar with perl yet so couldn't figure out how to run it manually or debug why it might not be called. Any ideas? Running Ubuntu 18.04 if it matters.Last modified on 2018-10-28 at 18:10
#15 by yorhel
2018-10-28 at 18:30
You're correct that the linked piece of code is responsible for updating the c_search column, but the way that code is triggered is perhaps not very obvious. That code is part of Multi, which is a separate process that runs in the background. It's not a crucial process for most development and testing, so I hadn't taken the time to make sure it works inside the Docker image yet.

In theory, the following should start it:
docker exec -ti vndb su -l devuser
cd /var/www
make multi-restart
But when I try that I'm getting a "Lost connection to PostgreSQL" which I can't explain at the moment.
#16 by alto
2018-10-29 at 19:58
Thanks, that makes a lot of sense and helped me fix it. It looks like Multi was using TCP to connect to Postgres which wasn't allowed. After fixing that and changing Multi to use the vndb_multi user to connect, everything works.

Reply

You must be logged in to reply to this thread.