Adding capabilities to Mac OS X Lion Server

UPDATE:

I never converted to Lion Server.  You can sortof see things unraveling in the middle of this post.  I’m taking another run at it now that Mountain Lion Server (now renamed back to OSX Server) is getting stable.  I sympathize with what Apple is trying to do.  If you’re kindof the power user in the office, the newer version of Server is much better for you.  But for those of us who were using the server to do slightly more complicated stuff, it’s been a long hard road.

I’ll write another post pretty soon that summarizes how I put stuff back into Mountain Lion Server.  It’s still not easy, but it’s going better — at least so far.  For now, just ignore the rest of this post.  It’s out of date, and it didn’t result in a working server.

Mikey

_________________________________________________________________________

This is another “scratchpad” post as I make the transition from Snow Leopard to Lion on my little family cloud server.  Here’s why the struggle is worth it for me;

  • Staying with the current release means Apple is updating my platform, which in turn means…
  • Better security/stability
  • Better compatibility with the iGadgets
  • Ease of use

The design philosophy for Server changed just a bit from Snow Leopard to Lion.  Lion Server is built on pretty much the same foundation, but the user-interface has been dramatically thinned out with the aim of making Server something that regular people could use.  I get that, and thing it’s a rational decision by Apple.  I was astounded to learn however that I’m in the “advanced user” category and lost some capabilities when this happened.  Who’da thunk it??  🙂

So I’ve got to go looking for ways to “put back” some of the things I use the server for.  My goal is to either find work-arounds within Lion Server or find bits and pieces of software that I can run on top of Lion to do those things.

This post will be the place where I post my findings — both about installing and configuring Lion, and solving the little work-around problems.  Should be fun.

Installation puzzlers

Running Lion in a VMWare virtual machine

Turns out that VMWare 4 brought in support for running instances of Lion in a virtual machine.  Kewl!  So I ran off and bought Lion Onna Stick (USB flash drive) from Apple, plugged it into my MacBook Pro, pointed VMWare Fusion at it, accepted the defaults, took a nap and when I came back I had me a Lion machine running on top of Snow Leopard.  Things to do differently from just accepting defaults;

  • Give the VM at least two cores in the CPU (runs a lot better — I may bump it to four the next time around).  Once Server is installed, my little Lion VM runs just dandy on the 2009 MacBook Pro — consumes about 5% of the CPU when idle.  Sweet.
  • When building Lion (not server, just Lion) pick a user/computer name that’s not a real personal type name — I ran into conflicts with my personal name in Open Directory because I’d already used it for the core Lion account.
  • Pay attention to networking — you’ll be using the Ethernet adapter a lot more rigorously than the default NAT configuration in VMWare — I set mine to go directly to the gateway router rather than using the default virtual-NAT.
  • Since we’re configuring the basis for a server here (especially if you want it to run Open Directory), this is the best time to get the DNS stuff sorted out.  I waited until later the first few times and the Server install vacuumed up a bunch of wrong-settings as a result.  I think I’ll do a little “Networking and DNS” section about all that.  Open Directory’s auto-configuration/startup process will break badly if DNS isn’t set up right.  I never figured out how to fix it after the fact — clean install with proper DNS was my path to success.
  • Take lots of snapshots of the VM.  The basic Lion install was pretty clean (except for the wrong-DNS stuff, see below), but I had to fall back to it several times before I got Server settled in properly (especially Open Directory).  The nice thing is that the App Store was quite happy to let me re-download the Server stuff and re-install it once I’d bought it.  I don’t know if there’s a limit, but I’ve re-installed Server on top of my clean Lion at least five times so far.  The word “Doh!” covers the reasons-why pretty well.

Networking and DNS for Lion Server

One of the things that really caught me was installing Lion Server behind an at-home gateway router.  In the past I’ve always been using a data-center router as the gateway and DNS was a no-brainer — just set up an A Record pointing at the server in DNS and go.  But home routers have a different job to do and those differences got pulled into the configuration of the server in ways that I wasn’t expecting.  Here are lessons-learned.

  • I’d never paid attention to the network name of my home router because in normal circumstances it doesn’t matter.  But since I am now using it as a gateway out to the “real” internet, it does.
  • My router thought it was in the “lan” domain — which is fine for a NAT-providing home router.  The trouble came when Lion Server pulled that domain into the name of the server when it talked to Lion during install.  Lion had in turn pulled in that “lan” domain through DHCP during install and built the computer-name with it (Mikes-Mac.lan or somesuch).  Again, this normally doesn’t matter, but that’s not a good name for a machine that is going to be put out on the public Internet.
  • My solution was to pound the real domain into the home router (CloudMikey.com in my case) before building Lion (yes Lion — don’t wait for the Server install — many headaches avoided).  That way all the computer-name bits and bobbins will have a real internet-routeable name instead of a non-routeable name.

Replacing Functionality

The good news about Lion Server is that it’s built on the same platform as all the earlier versions of Server.  The bad news is that the user interface has been redesigned with a different user in mind.  Not complaining, I get why they did this and it makes sense to me.  But I need to hunt around a bit to “add back” some of the tools that disappeared.  Here’s where I’ll take notes about that — my first pass will be based on scouring the Apple discussion-list for Lion Server and then I’ll see where I go from there.

Mail — Mail-forwarding and email-group accounts

My use of the mail server is pretty standard, but I have a few accounts which forward mail to a different address (mostly family members that retrieve their mail from their ISP’s server but want a consistent email address, or multiple people instead of just one).  I used the “Mail” tab in Workgroup Manager to do this on Snow Leopard, but that tab is missing in the Lion version of Workgroup Manager.

  • In Lion — build a filter using the webmail interface.  Once the account has been set up in the Workgroup Manager, log into the account with webmail and add filters that redirects messages to the downstream addresses.  One filter per address (rather than multiple addresses, separated by commas).  There’s a limit of 4 destinations per account, which is fine for me — most of mine are single destination forwarding accounts.  There’s a hack to expand that 4-destination limitation but I haven’t had to use it.

Mail — Hosting multiple domains for email

I use several domains for email.  Under Snow Leopard I would add them as as either Local Host Aliases or Virtual Domains in the Mail/Advanced/Hosting tab of Server Admin.  Doh!  They’re still there in the new version.  I was looking at Server rather than Server Admin.  Silly me.

Mail — Email aliases

These work the same as before — Workgroup Manager.

Web — SSL on sites

Initial post:

SSL encryption is pretty important to me, especially on web-based versions of wiki, mail, calendar, contacts, etc.  Don’t want people logging into those over an unencrypted connection, thank you very much.  So we gotta turn SSL on for some sites, but not all.

Argh.  I struggled with this for far too long. Did all kinds of fooling around with the files in the Apache “sites” folder, only to watch them get overwritten by Server each time I restarted it.  Worked all the way into the “readme” file in the Apache folder, on and on.  Terrible pain in the neck.  Nothing worked

Then I discovered the “Help” system in the Server app (not Server Admin, although the help system is fine there too).  SSL for virtual sites done in a different place.  Which Help told me.  Bah.  Went to the “Hardware/Server/Settings/SSL Certificate/Edit” menu, picked a certificate for the virtual site (and maybe restarted the web service) and it was set.  Does exactly the right thing too — when somebody goes to an SSL-enabled virtual site, they’re automatically redirected to the SSL version.

UPDATE 9-Jan:

Unfortunately, this returns to the “open issue, broken” status.  I’ve managed to wedge the Server app so that there are two states:

  • State 1 — everything turned off in the Server app including “web”
  • httpd daemon is running (sites respond to external requests, but with the /var/empty folder)
  • no functionality
  • relatively quiet logs (sample: Jan  9 01:05:32–Jan  9 05:05:31)
  • something odd going on with MySQL, probably unrelated)
  • Jan  9 01:06:29 server SubmitDiagInfo[4016]: Submitted shutdown stall report: file://localhost/Library/Logs/DiagnosticReports/ipfwloggerd,mysqld,sh_2012-01-01-080056_localhost.shutdownStall
  • something odd going on with xscertd (once an hour)
  • 1/9/12 6:05:24.632 AM sandboxd: ([6369]) xscertd(6369) deny job-creation
  • State 2 — “web” turned on, but NO SSL certificates assigned
  • httpd daemon is running (sites respond to external requests, but with the /var/empty folder)
  • no functionality
  • quiet logs — check logs around 6:52;28 AM for startup messages.  here are interesting ones;

1/9/12 6:52:28.713 AM xscertd: Starting xscertd/1.0.0 (MacOS X Server)
1/9/12 6:52:28.721 AM sandboxd: ([6723]) xscertd(6723) deny job-creation
1/9/12 6:52:31.176 AM servermgrd: servermgr_web: waiting for pid, file /private/var/run/httpd.pid.

  •  
  • State 3 — “web” turned on AND an SSL certificate is assigned
  • httpd daemon is NOT running (browser returns “problem loading page” and “unable to connect” errors
  • To get to this state — 1) shut down “web” in Server.app at 7:00:08 2) assign cert at 7:01:16 3) restart “web” at 7:03:46 4) shut off “web” again at 7:29:19 5) removed cert at 7:30:43
  • Here’s an extract of the interesting log messages:shut down “web” in Server app – 7:00:08Jan  9 07:00:08 server sandboxd[6807] ([6806]): xscertd(6806) deny job-creation
    Jan  9 07:00:09 server servermgrd[808]: servermgr_web: Disabling port forwarding for port 80
    Jan  9 07:00:11 server servermgrd[808]: servermgr_web: waiting for pid, file /private/var/run/httpd.pid.
    Jan  9 07:00:12 server servermgrd[808]: servermgr_web: Enabling port forwarding for port 80
    Jan  9 07:01:10 server CoreCollaborationServer[6852]: [main.m:103 40a280 +0ms] HTTP server listening at loopback:4444
    Jan  9 07:01:10 server com.apple.collabd[6852]: Jan  9 07:01:10 server.cloudmikey.com CoreCollaborationServer[6852] <Warning>: [main.m:103 40a280 +0ms] HTTP server listening at loopback:4444
    Jan  9 07:01:10 server com.apple.launchd[1] (com.apple.collabd[6852]): Tried to setup shared memory more than once
    Jan  9 07:01:10 server wikiadmin[6858]: Updating schema…
    Jan  9 07:01:10 server com.apple.collabd[6852]: 2012-01-09 07:01:10.231 wikiadmin[6858:307] Updating schema…
    Jan  9 07:01:10 server wikiadmin[6858]: Schema updates completed.
    Jan  9 07:01:10 server com.apple.collabd[6852]: 2012-01-09 07:01:10.235 wikiadmin[6858:307] Schema updates completed.
    Jan  9 07:01:15 server servermgrd[808]: servermgr_notification[I]: External configuration change detected, re-loading: c2s.xml
    Jan  9 07:01:15 server servermgrd[808]: servermgr_notification[I]: External configuration change detected, re-loading: Jan  9 07:01:17 server com.apple.launchd[1] (org.apache.httpd[6892]): Exited with code: 1
    Jan  9 07:01:17 server com.apple.launchd[1] (org.apache.httpd): Throttling respawn: Will start in 10 seconds
    Jan  9 07:01:17 server servermgrd[808]: servermgr_notification[N]: jabberd service startup completed.
    Jan  9 07:01:18 server jabberd_notification/router[6886]: [127.0.0.1, port=57627] connect
    Jan  9 07:01:18 server com.apple.APNBridge[6901]: http server appears to have started
    Jan  9 07:01:18 server com.apple.APNBridge[6901]: Connected to XMPP server
    Jan  9 07:01:18 server jabberd_notification/router[6886]: [127.0.0.1, port=57627] authenticated as apn.server.cloudmikey.com
    Jan  9 07:01:18 server jabberd_notification/router[6886]: [apn.server.cloudmikey.com] online (bound to 127.0.0.1, port 57627)
    Jan  9 07:01:18 server jabberd_notification/router[6886]: [127.0.0.1, port=57628] connect
    Jan  9 07:01:18 server jabberd_notification/router[6886]: [127.0.0.1, port=57628] authenticated as pubsub.server.cloudmikey.com
    Jan  9 07:01:18 server jabberd_notification/router[6886]: [pubsub.server.cloudmikey.com] online (bound to 127.0.0.1, port 57628)
  • restart “web” at 7:03:46
  • Jan  9 07:03:09 server xscertd-helper[6808]: idle timer triggered, exiting
  • Jan  9 07:03:46 server servermgrd[808]: servermgr_web: enabling
    Jan  9 07:03:48 server sandboxd[6979] ([6978]): xscertd(6978) deny job-creation
    Jan  9 07:03:49 server servermgrd[808]: servermgr_web: Disabling port forwarding for port 443
    Jan  9 07:03:50 server servermgrd[808]: servermgr_web: waiting for pid, file /private/var/run/httpd.pid.
    Jan  9 07:03:55: — last message repeated 3 times —
    Jan  9 07:03:55 server servermgrd[808]: servermgr_web: Enabling port forwarding for port 443
    Jan  9 07:03:55 server servermgrd[808]: servermgr_web: Cannot confirm Apache was started; missing or invalid pid file
    Jan  9 07:07:25 server xscertd-helper[6980]: idle timer triggered, exitingshut off “web” again at 7:29:19
    Jan  9 07:29:19 server servermgrd[808]: servermgr_web: Disabling port forwarding for port 443
    Jan  9 07:29:20 server servermgrd[808]: servermgr_web: waiting for pid, file /private/var/run/httpd.pid.
    Jan  9 07:29:20 server com.apple.launchd[1] (org.apache.httpd[7792]): Exited with code: 1
    Jan  9 07:29:20 server com.apple.launchd[1] (org.apache.httpd): Throttling respawn: Will start in 10 seconds
    Jan  9 07:29:21 server servermgrd[808]: servermgr_web: waiting for pid, file /private/var/run/httpd.pid.
    Jan  9 07:29:25: — last message repeated 3 times —
    Jan  9 07:29:25 server servermgrd[808]: servermgr_web: Enabling port forwarding for port 443
    Jan  9 07:29:25 server servermgrd[808]: servermgr_web: Cannot confirm Apache was started; missing or invalid pid fileremoved cert at 7:30:43
    Jan  9 07:29:19 server servermgrd[808]: servermgr_web: Disabling port forwarding for port 443
    Jan  9 07:29:20 server servermgrd[808]: servermgr_web: waiting for pid, file /private/var/run/httpd.pid.
    Jan  9 07:29:20 server com.apple.launchd[1] (org.apache.httpd[7792]): Exited with code: 1
    Jan  9 07:29:20 server com.apple.launchd[1] (org.apache.httpd): Throttling respawn: Will start in 10 seconds
    Jan  9 07:29:21 server servermgrd[808]: servermgr_web: waiting for pid, file /private/var/run/httpd.pid.
    Jan  9 07:29:25: — last message repeated 3 times —
    Jan  9 07:29:25 server servermgrd[808]: servermgr_web: Enabling port forwarding for port 443
    Jan  9 07:29:25 server servermgrd[808]: servermgr_web: Cannot confirm Apache was started; missing or invalid pid file
  • 1/9/12 6:52:37.981 AM com.apple.SecurityServer: setupThread failed rcode=-2147418111

UPDATE 12-Jan:

The road to recovery.  I spoke with Apple Support and worked my way up to a Tier-2 support person who helped me out a lot.  He gave me a bunch of great pointers which I’ll post here as I use them.  He was very careful to point out that some of this is for experienced folks only, your mileage may vary, if you break it you bought it and some of this may result in something that’s so broken that it falls outside the normal free telephone support.  Be careful!

The problem seems to be caused by the way I set the server up.  Y’see, I built the server at the farm and then moved it to the data center.  So the IP address changed.  That IP address gets “baked in” to a bunch of things, and especially the SSL certificate that gets created when the server is first configured.  Moving the server to a new IP-address puts it out of sync with the information in the certificate and that’s very likely what’s causing the problem.

Step 1 — Set the Web server back to defaults.

Here’s a link to the page in the Advanced Administration guide for Lion Server — https://help.apple.com/advancedserveradmin/mac/10.7/#apd163efc3a-1465-4a44-ad2d-c76094144512

My sequence of steps was this;

  • Toggle off all the services in the Server application and turned off the SSL cert
  • Run “sudo serveradmin command web:command=restoreFactorySettings” (omit the quotes) repeatedly while at the same time watching the logs in Server.  The command failed several times because it couldn’t find copies of various default versions of config files in the /var/apache2/sites/ folder.  Fortunately, I have backup copies of those files so I just replaced them one at a time until the command ran to the end successfully.

Step 2 — Create a new SSL cert

  • Created a new SSL certificate in the Server application (Hardware/YourServerName/Settings/”Edit” SSL certificate/select the “gears” dropdown/select “manage certificates”/click the “+” button to add a new certificate/select “create a certificate identity”/accept the defaults/)

Step 3 — Cycle the server and cross fingers

  • Rebooted the server
  • Waited for the logs to quiet down
  • Started the Web service and watched it create it’s config files in the apache2/sites folder — logs were still quiet
  • Assigned the newly-created SSL cert (I wish I could delete the old one but I can’t) — logs are still quiet
  • Turned on the Wiki service — logs are still quiet
  • So far so good!  I think I’ll leave things like this for a while before adding back the other services and the custom web sites.  More updates to follow.

Web — MySQL

Lion switched from MySQL to PostGres (rumbles of ORACLE lawsuits no doubt) so I’ve got to start running a “real” version of MySQL so that all the little WordPress sites continue to function.

  • Hm.  MySQL only supports OS X through Snow Leopard — looks like we’re kinda out here on our own.  <shrug, what could go wrong?>
  • Downloads are here  – http://dev.mysql.com/downloads/mysql/ (roll down to the DMG file — way easier install)
  • Installation instructions are here – http://dev.mysql.com/doc/mysql-macosx-excerpt/5.5/en/macosx-installation.html
  • Documentation is here – http://dev.mysql.com/doc/index.html (haven’t used it yet)
  • PHP needs to be tweaked – https://support.apple.com/kb/HT4844 (I only did the “change-sockets to /tmp/mysql.sock” thingy)
  • Installed Sequel Pro (http://www.sequelpro.com/) and tested the installation by creating and dropping a database.

Web – loading up a WordPress site

Let’s see how much of the Lion stuff I can use…

  • Point a domain at the server (an A record in DNS)
  • Create a new site in the Server app (using the same domain name)
  • Copy in WordPress files (download them from http://www.wordpress.org)
  • Give ownership to _www user (CD into the folder *above* the folder for the site is and type “sudo chown _www your-site’s-foldername” in Terminal)
  • Transmit ownership to all files in the folder (Finder/Get info/Unlock/Permissions/Apply to enclosed items)
  • Create a database (I use Sequel Pro — create an empty database and a user that has full rights to the database)
  • Create the wp-config.php that points at the database

Web — point multiple URLs at the same site

I don’t do this often, but sometimes I point more than one variant of a domain at a site.

  • Lion way — create an addition site in the Server app — new URL, pointed at the same content directory as the first site.  Works fine  Ooops…  things get sticky when doing this — I wound up with a bunch of Apache site configuration files, and thus the opportunity of conflicts.  Better way…
  • Set the site up in the Server app with *just* the domain name (leave the “www” variant for the next step)
  • Edit the site configuration (file etc/apache2/sites/ip-address-stuff_port-number_domain-name.conf) and add ServerAlias records at the very bottom of the file, just before the closing </VirtualHosts> entry.
  • Like this:
  • ServerAlias www.example.com
  • ServerAlias good.example.com
  • ServerAlias bad.example.com
  • Restart the web server (and clear the browser cache) to check

Web — redirects

I like to throw redirects into sites from time to time.  In Snow Leopard, this was easily done through Server Administrator but that’s gone in Lion.  Adding them into the Apache files isn’t too bad though.  Here’s how.

  • Open the site file (etc/apache2/sites/ip-address-stuff_port-number_domain-name.conf — I like the TextWrangler editor for this kind of stuff)
  • Insert a section that looks like this (I lifted this from my Haven2.com file on the Snow Leopard file and stuck it into my Dissembling.com test site);

<IfModule mod_alias.c>
Redirect temp “/rss.xml” “http://feeds.feedburner.com/Haven”
</IfModule>

  • Only need one set of bracketed “IfModule” statements, and stick in as many “Redirect temp” statements as needed.
  • I’ll probably just copy these sections over from their files on the Snow Leopard server and see how they work out.
  • Restart the web server (toggle Web off and back on in the Server app)

Web — separate log files

Some of my domains get a lot of traffic and it’s handy to be able to strain out their stuff into a separate log file.  Not a show-stopper but handy.  Once again, the site files in Apache seem to be the place to do this.

  • Open the site file (etc/apache2/sites/ip-address_port-number_domain-name.conf)
  • Change the CustomLog and ErrorLog statements to point at a unique file rather than the default
  • Restart the web server
  • Check to make sure things are working by looking in var/log/apache2 for the new files after the restart
  • Best to open the log files with the Console app — lots easier to read the files (and get real-time updates)

Web — rotate log files

I like to have the log files break themselves up into weekly chunks so i can go clear out the old ones every once in a while.  In Snow Leopard, this was easy — just tick the little box and it did it.  Lion makes me work harder.

  • Open the site file (etc/apache2/sites/ip-address_port-number_domain-name.conf)
  • Change the CustomLog from this:

CustomLog “/var/log/apache2/example_access_log”

  • To this:

CustomLog ‘|/usr/sbin/rotatelogs “/var/log/apache2/example_access_log” 604800 -360’ “%h %l %u %t \”%r\” %>s %b”

  • Change the ErrorLog from this:

ErrorLog “/var/log/apache2/example_error_log”

  • To this:

ErrorLog ‘|/usr/sbin/rotatelogs “/var/log/apache2/example_error_log” 604800 -360’

One wonders if making these changes to the default version of the configuration file would drive this stuff in automagically.  Might just research that some day.

Web — permalinks in WordPress sites

WordPress has the ability to change the format of the URLs for posts and pages from the ugly PHP link to a prettier “permalink” structure.  Apache needed to be tweaked in Snow Leopard to make this work right, and it still does in Lion.  Here’s how.

  • The etc/apache2/httpd.conf file needs to be changed (only once, the first time through) so that the “AllowOverride” statement in the “/Library/WebServer/Documents/” section reads “AllowOverride All” (there are several AllowOverride statements in httpd.com — pay attention to which one is being changed).  Note: I’m not sure this step is really required — my testing was a little horked up and I’m too lazy to repeat it to verify
  • Open the site file (etc/apache2/sites/ip-address_port-number_domain-name.conf)
  • Change the statement “AllowOverride None” to “AllowOverride All” in the “Directory” section
  • Create a .htaccess file in the site directory (use Terminal, CD to the site directory, “sudo touch .htaccess”)
  • Change ownership of the .htaccess file to the “_www” user (“sudo chown _www .htaccess”) — this lets WordPress modify the .htaccess file with the permalink rules.
  • Restart the web service in the Server app
  • When all else fails (I had a heck of a time getting the server to write the .htaccess file correctly — although restarting Finder [Apple-menu/Force-quit…/Finder/Restart] may have cured that problem) I manually edit the .htaccess file.   Here’s the code that needs to be in it:
# BEGIN WordPress
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteBase /
RewriteRule ^index\.php$ - [L]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . /index.php [L]
</IfModule>
# END WordPress

Conclusion

Well, none of this is real tough — so I think I’m about ready to start moving stuff over to the Lion environment.  I’ll probably wind up running it under a virtual machine until I’ve converted everything.  Then I’ll explore moving it out of the virtual machine back into a native Lion install on my tiny little server.  Or maybe not.  That’s for another day.