Homestead but on FreeBSD - A Closer Look

The why of the Vagrantfile

Caleb Marble

2017-07-03

What is this?

This is an introduction to Homestead but with FreeBSD as a host system. I wanted something leaner than Homestead to develop on and needed my development environment to match my staging and production environments (since both run FreeBSD).

I chose FreeBSD-11.1-BETA3 as a FreeBSD version because 12-CURRENT was unstable at the time I made the decision. 10.3 was not considered due to its fast approaching end of life and 11.0-STABLE had a problem with the vboxguest.ko module which caused the virtual machine to enter a reboot loop. I also did not want to use the bento/FreeBSD versions due to them including a lot of packages that are not in the base system. It would also defeat the choice for a leaner development environment that's faster to get up and running due to the ~1 GB VirtualBox image size (bigger than Homestead, which is only ~850 MB). This will change to 11.1-STABLE when that is officially release. Until then I'm keeping my github branch at 1.0.0-BETA.

I chose not to release a hashicorp box because the Vagrantfile and accompanying scripts do everything for you. I feel this approach is more appealing because it allows you, the developer, to see exactly what was done to a box and how to fix something if it goes wonky, and since it's directly from the FreeBSD release engineers it's incredibly stable and well supported. Since there's no trade-off in convenience for the developer I felt that developing a box would be useless. However, if there is enough demand for it I will consider releasing a Vagrant box image that's based off the FreeBSD stable train.

Introduction to the Vagrantfile

Here I will attempt to go line by line to state my reasoning on why I did certain things. Hopefully this will prove to be a lasting documentation on how to further maintain this virtual machine in the future should I no longer be able to update it myself.

Temporary Directories

    config.vm.box_check_update = true
    config.ssh.shell = "sh"
    settings["keys"].each do |key|
        if File.exists? File.expand_path(key)
            config.vm.provision "shell" do |s|
                s.privileged = false
                s.inline = "echo \"$1\" > /home/vagrant/.ssh/$2 && chmod 600 /home/vagrant/.ssh/$2"
                s.args = [
                    File.read(File.expand_path(key)),
                    key.split('/').last
                ]
            end
        end
    end

This code is responsible for adding any keys you specified to the vagrant box. This was copied directly from the official Homestead Vagrantfile.

    config.vm.network :private_network, ip: "192.168.10.10"
    config.vm.base_mac = "0800270CA7EB"

I chose not to give the user the option to change the IP address due to how I mount the SMB shares for Windows users. If this conflicts with an IP address on your network and you wish to change it be sure to change any instance of '192.168.10.1' in the Vagrantfile to the IP of your host machine and any instance of '192.168.10.10' to the new machine IP.

    config.vm.provision "shell" do |s|
        s.name = "Preemptively creating tmp directories."
        s.inline = "rm -rf /tmp/init && mkdir -p /tmp/init && chown vagrant:vagrant /tmp/init"
    end

This is done so that all files in the Includes and Config directories can be uploaded safely to the remote site.

I have to chown the newly created folder because the folders are created with the owner of root:root and config.vm.provision "file" will scp the files as the vagrant user.

File and Script Provision

    config.vm.provision "file", source: vagrant_dir + "/Config/", destination: "/tmp/init/"
    config.vm.provision "file", source: vagrant_dir + "/Scripts/", destination: "/tmp/init/"

Uploads directories to newly created vagrant box. Expected folder layout is '/tmp/init/Config/...' and '/tmp/init/Scripts/...'

    config.vm.provision "shell" do |s|
        s.name = "Provisioning scripts"
        s.inline = "mv /tmp/init/Scripts/* /usr/local/sbin/ && chmod +x /usr/local/sbin/*"
    end

Moves all uploaded scripts to '/usr/local/sbin' and makes sure they're executable.

Be careful to not name your own scripts anything that would already exist in '/usr/local/sbin', as they will be replaced.

Hosts file

    config.vm.provision "shell" do |s|
        s.name = "Provisioning clean hosts file"
        s.inline = "mv /tmp/init/Config/hosts /etc/hosts"
    end

This will overwrite the existing /etc/hosts file. While this does give the user the ability to manually add anything they want to this file, it's more intended to overwrite the /etc/hosts file on every vagrant provision command so that a later provision script can append to it without worrying about adding duplicates (which would be a problem if the default IP address ever changes, making the first result from /etc/hosts be the old IP address instead of the new one which would be added to the bottom of the file).

Project Folder Synchronization

    if settings.include? 'folders'
      settings["folders"].each do |folder|

        config.vm.provision "shell" do |s|
            s.name = "Creating #{folder["to"]}"
            s.inline = "mkdir -p #{folder["to"]}"
        end

        mount_opts = []

        if (folder["type"] == "nfs")
            mount_opts = folder["mount_options"] ? folder["mount_options"] : ['actimeo=1']
        elsif (folder["type"] == "smb")
            mount_opts = folder["mount_options"] ? folder["mount_options"] : ['vers=3.02', 'mfsymlinks']
        end

        # For b/w compatibility keep separate 'mount_opts', but merge with options
        options = (folder["options"] || {}).merge({ mount_options: mount_opts })

        # Double-splat (**) operator only works with symbol keys, so convert
        options.keys.each{|k| options[k.to_sym] = options.delete(k) }

        if (folder["type"] == "smb")
            config.vm.synced_folder folder["map"], folder["to"],
                type: folder["type"],
                smb_id: File.basename(folder["map"]),
                smb_username: settings["smb_info"]["smb_username"],
                smb_password: settings["smb_info"]["smb_password"],
                **options
        else
            config.vm.synced_folder folder["map"], folder["to"], type: folder["type"] ||= nil, **options
        end

        if folder.include? 'hostname'
            config.vm.provision "shell" do |s|
                s.name = "Adding hostname for #{folder["hostname"]}"
                s.inline = "printf '192.168.10.10\t#{folder["hostname"]}\n' | tee -a /etc/hosts"
            end
        end

        # Bindfs support to fix shared folder (NFS) permission issue on Mac
        if Vagrant.has_plugin?("vagrant-bindfs")
          config.bindfs.bind_folder folder["to"], folder["to"]
        end
      end
    end

This is where a lot of the folder synchronization magic happens. Each folder entry is supposed to be the top level for a single laravel application.

You have to first make sure the folder is created inside the box when mounting under a Windows SMB environment.

The SMB shares are mounted like this to make sure I can manually mount them later through /etc/fstab. By default Vagrant will assign a share a random name and I needed something consistent. I considered not including the SMB folder config so that vagrant up on Windows wouldn't complain, but that turns out to be a whole new headache on it's own. I'd either have to write a ton of fragile scripts around this feature for creating/destroying SMB shares and mounting inside of the Vagrant box or create a Vagrant plugin giving BSD hosts SMB mount capabilities.

Adding the hostname of the mounts to the /etc/hosts inside the Vagrant box is very important as that's how Apache can tell which site you want from the hostname you use to connect to the box. And it's easier to type 'cool-crm.app' into a browser than to type '192.168.10.10'.

Everything else about this function is the default Homestead.yaml magic.

    config.vm.synced_folder ".", "/vagrant", disabled: true

This is required because the default synced_folder is not nfs by default and causes a error in the provision step.

Vagrant Settings

    config.vm.provider "virtualbox" do |vb|
        vb.name = "pwdev.app"
        # 640K ought to be enough for anybody.
        vb.memory = settings["memory"] ||= "2048"
        vb.cpus = settings["cpus"] ||= "1"
        vb.gui = settings["gui"] ||= false
    end

Increase the values found in ~/.config/Vagrant.yaml if you find you need a beefier virtual machine.

    config.vm.boot_timeout = 600 # the default `300' gave me timeouts

Comment for this line says it all. There's a timeout because the initial setup when doing vagrant up installs sudo and some other packages and I was on a supremely slow internet connection at the time.

Timezone

    config.vm.provision "shell" do |s|
        s.name = "Setting Timezone"
        s.inline = "cp $@"
        s.args = [
            "/usr/share/zoneinfo/" + settings["timezone"] ||= "America/Chicago",
            "/etc/localtime"
        ]
    end

When doing a FreeBSD install for the first time you want to run tzsetup(8) if you want to have log file timestamps and other things to be in your local time zone. However, it uses dialog options we can't script because it'll come back with "cannot open tty-output". The manpage mentions that you should be able to get around this in scripts by mentioning the exact time zone file, but doing so still brings up a confirmation dialog box which also requires user input. As a workaround I have simply cp'd the file to the proper location.

Why is "America/Chicago" the default time zone? Because that's where I am and I'm exceedingly lazy about these things. To change it make sure you have a proper time zone name. You can find them in '/usr/share/zoneinfo' on any standard BSD system. Do not include the full file path, I did that for you.

Host Packages

    config.vm.provision "shell" do |s|
        s.name = "Installing packages"
        s.inline = "pkg update && FETCH_RETRY=10 pkg upgrade --yes && FETCH_RETRY=10 pkg install --yes $@"
        s.args = [
        ...
        ]

Installs packages to the host system. You can find requirements for specific things inside comments for each section.

I had to use the FETCH_RETRY=10 setting to increase the default retry from 3 to 10. This was due to the terrible internet I had while working on this, that even after 3 attempts the package would still sometimes fail to download.

    config.vm.provision "shell" do |s|
        s.name = "Install grunt"
        s.inline = "npm install -g npm grunt-cli"
    end

I use grunt to build the static resources for our websites. If you don't use grunt at all or would prefer to use grunt outside the vagrant machine feel free to remove this.

Host Environment

    $environment = <<-SCRIPT
        :>/etc/motd
        printf 'fortune futurama\necho\n' > /etc/profile
        chsh -s /usr/local/bin/bash vagrant
        mv /tmp/init/Config/login.conf /etc/login.conf
        chown root:wheel /etc/login.conf
        cap_mkdb /etc/login.conf
    SCRIPT

    config.vm.provision "shell" do |s|
        s.name = "Setting up environment"
        s.inline = $environment
    end

First we clear the /etc/motd file so when we vagrant ssh into our box we don't get a wall of text. Then we change the shell for the vagrant user to bash (bash isn't required for anything, I just prefer it. Change/remove as you please).

Finally we copy our local login.conf file to /etc/login.conf and run cap_mkdb(1) to build the database that is actually loaded by the FreeBSD OS on boot. The default login.conf file in the Config/ folder only enables UTF-8 (required for UTF-8 mysql tables).

Host Services & Config

    $services = <<-SCRIPT
        sysrc hostname="pwdev"
        sysrc apache24_enable="YES"
        sysrc mysql_enable="YES"
        sysrc rpc_lockd_enable="YES"
        sysrc rpc_statd_enable="YES"
    SCRIPT

    config.vm.provision "shell" do |s|
        s.name = "Enabling services"
        s.inline = $services
    end

We enable the basic services for the system. See: sysrc(8) for more info. It's great, everyone should know about it.

rpc_lockd_enable and rpc_statd_enable are required for exclusive locking on nfs. I must admit I do not fully understand enough about file systems to be able to answer why this is needed, but I got it from here: https://github.com/thephpleague/flysystem/issues/445.

The FreeBSD Handbook on locking only states that, "Some applications require file locking to operate correctly".

    config.vm.provision "shell" do |s|
        s.name = "Configuring apache"
        s.inline = 'mv /tmp/init/Config/Includes/*.conf /usr/local/etc/apache24/Includes/'
    end

Move any includes file in the Config/Includes/ directory into the apache24 Includes folder. Necessary for each site that has a hostname defined. You can find examples in the Config/Includes/ folder.

MySQL

    $mysql = <<-SCRIPT
        mv /tmp/init/Config/my.cnf /usr/local/etc/mysql/my.cnf
        service mysql-server start
        /usr/local/bin/mysql --user=root --connect-expired-password --password="$(tail -1 /root/.mysql_secret)" -e "ALTER USER 'root'@'localhost' IDENTIFIED BY 'secret';" 2>/dev/null
        /usr/local/bin/mysql --user=root --password=secret -e "FLUSH PRIVILEGES;" 2>/dev/null
    SCRIPT

    config.vm.provision "shell" do |s|
        s.name = "Provisioning mysql"
        s.inline = $mysql
    end

    if settings.include? 'mysql'
        settings["mysql"].each do |db|
            config.vm.provision "shell" do |s|
                s.name = "Creating database: #{db["database"]} controlled by #{db["user"]}"
                s.inline = "/usr/local/bin/mysql --user=root --password=secret -e \"" \
                "CREATE DATABASE IF NOT EXISTS #{db["database"]};" \
                "GRANT ALL ON #{db["database"]}.* TO '#{db["user"]}'@'%' identified by '#{db["pass"]}';" \
                "FLUSH PRIVILEGES;\""
            end
        end
    end

So, funny story. One of the projects I worked on called so many database queries that it was required by the original developers to run a mysql instance on the local machine; because otherwise the millisecond latency for each SQL command would add up to the point where every page load took upwards of 10 seconds to load. I've included the mysql service and the ability to configure each database with it's own user and password for each project for people in similar situations. See the default Vagrant.yaml file for an example of a database config with a password.

If you have a remote staging or development database you want to have your applications hit, feel free to remove this section and the sysrc command above that enables the mysql daemon. You can also remove the mysql-server package that's install in the host machine.

PHP Imprisoned

    $php_jails_init = <<-SCRIPT
        mv /tmp/init/Config/jail.conf /etc/jail.conf
        sysrc jail_enable="YES"
        umount -a -F /etc/fstab
        umount -a -F /etc/fstab.pkg
        :>/etc/fstab.pkg
        service jail stop
        chflags -R noschg /jails
        rm -rf /jails
    SCRIPT

    config.vm.provision "shell" do |s|
        s.name = "Preparing php jails"
        s.inline = $php_jails_init
    end

Oh boy! I get to talk about jails. Jails are so awesome they get their own FreeBSD handbook chapter. For a very quick rundown on the basics of Jails please see Papers We Love: Jails and Zones (then go on a Bryan Cantrill binge on youtube because everything he does is great).

In laymen's terms a jail is a virtualized instance of FreeBSD that doesn't have a kernel (it uses the host's kernel). They're incredibly lightweight (think chroot but with tons of security built in) and perfect for isolating applications; which we need to do here because I needed to work on both php56 and php70 apps and didn't want to spin up a vagrant machine for each. Also php-fcgi would refuse to work with multiple versions of php on the same machine. In fact, in FreeBSD, attempting to install both versions will cause pkg to ask you if you want to remove the other. Rather than forcing the installation of both and having a bad time(tm) (or worse, manually compiling both versions into separate install directories) I opted to make a jail for each version of php.

First I copy any new configuration changes to the /etc/jail.conf file and enable the jail service (if it isn't already). Then I stop the jail service if it's running and unmount any of the mounted file systems. On the first vagrant up this is useless, but if running vagrant provision after the initial install it is imperative to unmount all directories to make sure any project folders are not deleted.

I then completely remove all jails from the system if there are any and create the future jails' directories.

    config.vm.provision "shell" do |s|
        s.name = "Backing up /etc/fstab if backup doesn't exist"
        s.inline = "test ! -f /etc/fstab.release && cp -v /etc/fstab /etc/fstab.release || :"
    end

    config.vm.provision "shell" do |s|
        s.name = "Restoring /etc/fstab from backup"
        s.inline = "cp -v /etc/fstab.release /etc/fstab"
    end

Here I make sure I don't clobber the /etc/fstab file with multiple lines of the same mount command.

    jails.each do |jail|
        config.vm.provision "shell" do |s|
            s.inline = "mkdir -p $@"
            s.args = [
                "/jails/#{jail}/usr/local/etc",
                "/jails/#{jail}/var/cache/pkg"
            ]
        end

        config.vm.provision "shell" do |s|
            s.name = "Creating #{jail} jail"
            s.binary = true
            s.env = {
                "DISTRIBUTIONS" => "base.txz lib32.txz",
                "BSDINSTALL_DISTDIR" => "/usr/freebsd-dist",
                "BSDINSTALL_DISTSITE" => jail_dist_site
            }
            s.args = "/jails/#{jail}"
            s.path = vagrant_dir + "/Scripts/jail.sh"
        end

        # Used to keep the pkg cache in one place. Make re-provisioning much faster.
        config.vm.provision "shell" do |s|
            s.inline = "printf '%s %s %s %s %s %s\n' $@ | tee -a /etc/fstab | tee -a /etc/fstab.pkg"
            s.args = [
                "/var/cache/pkg",
                "/jails/#{jail}/var/cache/pkg",
                "nullfs", "rw", "0", "0"
            ]
        end
    end

    config.vm.provision "shell" do |s|
        s.inline = "service jail start"
    end

This is where the jails are actually created. jail.sh is a script I created to automate the install of a FreeBSD jail using a minimal configuration It's based off the official FreeBSD script used to install jails with a lot of stuff ripped out and outright changed to prevent dialog boxes from interrupting the install. In this case I knew quite a bit about how the setups should be installed so I was able to remove many of the standard bsdinstall(8) dialog boxes. You can check the jail.sh script out but what it boils down to is: download the base.txz and lib32.txz files from the distribution site, check their checksums, extract them into the jail directories, and copy over some local configurations. Many of the manual stuff I had to write for that script was mostly getting around the bsdinstall(8) dialog boxes. I'll be updating this file and documentation if I ever find a better way to do this. As of now it's stable and fast.

I then add the package cache folder to the jail mounted as nullfs. I use a separate fstab file so that I can mount and unmount them manually as needed. Normally this would be a useless thing to do but in this case it makes running vagrant provision multiple times very fast as it does not need to download PHP and all it's dependencies each time.

SMB Magic

    if settings.include? 'smb_info'
        config.vm.provision "shell" do |s|
            s.name = "Adding smb_username to /etc/nsmb.conf"
            s.inline = "printf '[%s]\naddr=%s\n[%s:%s]\npassword=' $@ > /etc/nsmb.conf"
            s.args = [
                ENV['COMPUTERNAME'].upcase,
                "192.168.10.1",
                ENV['COMPUTERNAME'].upcase,
                settings["smb_info"]["smb_username"].upcase
            ]
        end

        config.vm.provision "shell" do |s|
            s.name = "Adding smb_password to /etc/nsmb.conf"
            s.inline = "smbutil crypt $@ >> /etc/nsmb.conf"
            s.args = settings["smb_info"]["smb_password"]
        end
    end

This is my workaround for Vagrant not being able to mount SMB folders inside of a BSD box. There's a closed issue here: https://github.com/mitchellh/vagrant/issues/5432 which goes into more detail and why it wasn't added to Vagrant. If I (or someone else) find the time to add this feature I'll be able to remove this and rely only on Vagrant to mount the folders.

Fun Fact: it took me half a day to figure out that the hostname and the username must be IN ALL CAPS for /etc/fstab to be able to properly mount the folders on the machine. In my defence the FreeBSD manpage for nsmb.conf(5) did not explicitly say the values needed to be uppercase to work. All of my thanks goes to Sean G from the FreeBSD forums for posting his solution. Thanks, Sean.

If you are going to use SMB shares you must include your Windows credentials in %HOMEDRIVE%%HOMEPATH%\.config\Vagrant.yaml. This is used to automatically mount the appropriate folders without needing a username or password on every `vagrant up`.

NullFS Mounting our Projects into the Appropriate Jail

    if settings.include? 'folders'

        settings["folders"].each do |folder|
            site_folder = folder["to"]
            site_name = File.basename(site_folder)
            php_version = folder["phpver"]

            config.vm.provision "shell" do |s|
                s.inline = "mkdir -p $@"
                s.args = "/jails/#{php_version}/sites/#{site_name}"
            end

            # Custom mounts for samba shares. Required until Vagrant can
            # understand that FreeBSD can mount smb shares out of the box.
            # See: https://github.com/mitchellh/vagrant/issues/5432
            if (folder["type"] == "smb")
                config.vm.provision "shell" do |s|
                    s.name = "Appending samba shares to /etc/fstab"
                    s.inline = "printf '//%s@%s/%s /home/vagrant/%s smbfs rw,-N 0 0\n' $@ >> /etc/fstab"
                    s.args = [
                        settings["smb_info"]["smb_username"],
                        ENV['COMPUTERNAME'].upcase,
                        File.basename(folder["map"]),
                        File.basename(folder["map"]),
                    ]
                end
            end

            # ,late is used here to make sure the folders are mounted with the
            # host machine before they're mounted inside the jails, otherwise the
            # jail folders will be empty.
            config.vm.provision "shell" do |s|
                s.name = "Adding project folders to fstab as nullfs jail mounts"
                s.inline = "printf '%s %s %s %s %s %s\n' $@ >> /etc/fstab"
                s.args = [
                    "/home/vagrant/#{site_name}",
                    "/jails/#{php_version}/sites/#{site_name}",
                    "nullfs", "rw,late", "0", "0"
                ]
            end
        end
    end

Here I'm adding appropriate folders to /etc/fstab. You'll notice that the nullfs options are rw,late. This is so that the nullfs mount happens after any shares with the host machine are safely mounted.

From the fstab manpage:

     If the option ``late'' is specified, the file system will be automati-
     cally mounted at a stage of system startup after remote mount points are
     mounted.  For more detail about this option, see the mount(8) manual
     page.

Installing PHP into the new Jails

    config.vm.provision "shell" do |s|
        s.name = "Mounting folders before pkg install."
        s.inline = "mount -a -F /etc/fstab.pkg"
    end

    jails.each do |jail|
        config.vm.provision "shell" do |s|
            s.name = "Installing packages inside #{jail} jail"
            s.inline = "pkg -j #{jail} update && FETCH_RETRY=10 pkg -j #{jail} upgrade --yes && FETCH_RETRY=10 pkg -j #{jail} install --yes $@"
            s.args = [
            ...
            ]
        end

        if jail == "php56"
            config.vm.provision "shell" do |s|
                s.name = "Installing php56 specific packages inside php56 jail"
                s.inline = "pkg -j #{jail} update && FETCH_RETRY=10 pkg -j #{jail} upgrade --yes && FETCH_RETRY=10 pkg -j #{jail} install --yes $@"
                s.args = [
                    "#{jail}-mysql"
                ]
            end
        end

        config.vm.provision "shell" do |s|
            s.inline = "mv /tmp/init/Config/#{jail}-fpm.conf /jails/#{jail}/usr/local/etc/php-fpm.conf"
            s.args = [
                "/tmp/init/Config/#{jail}-fpm.conf",
                "/jails/#{jail}/usr/local/etc/php-fpm.conf"
            ]
        end

        config.vm.provision "shell" do |s|
            s.inline = "jexec #{jail} sysrc php_fpm_enable=\"YES\""
        end
    end

I make sure to mount the package cache into the jail before installing any packages. This makes future vagrant provisions faster.

Then I simply install the packages and the php-fpm.conf files you can find in the Config directory, followed by making sure the php-fpm service is enabled on the start of the jail.

Conclusion

That's it. Honestly this document is a little underwhelming for about two weeks worth of off-and-on work (so it goes). It is my hope that I included enough documentation here for the average BSD user to understand. If you have an issue with Vagrant not properly bringing up the box please add an issue to the GitHub Project.

Special thanks goes to everyone in IRC who helped me with this endeavor; and to all of you who post your solutions to the problems I ran into around the web in various places.


View Comments | Back to Blog | Back to Home