Set up the webstack repository and install amp packages for convenience (second step not required, but makes life easier):
$ pfexec pkg set-authority -O http://pkg.opensolaris.org/webstack/ webstack
$ pfexec pkg install amp amp-dev
Install Passenger, install and unpack nginx's sourcecode :
$ pfexec pkg install nginx
$ gunzip -c /opt/webstack-source/nginx/nginx-0.6.35.tar.gz | tar xvf -
On or around line 267 of /var/ruby/1.8/gem_home/gems/passenger-2.2.8/bin, replace extra_args = "" with the following, also adding this to line 292 before return extra_args:
extra_args = "--user=webservd --group=webservd --with-debug --prefix=/usr/nginx/ --sbin-path=/usr/nginx/sbin/nginx --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/nginx/logs/error.log --http-log-path=/var/nginx/logs/access.log --pid-path=/var/nginx/logs/nginx.pid --http-proxy-temp-path=/var/nginx/proxy_temp --http-fastcgi-temp-path=/var/nginx/fastcgi_temp --http-client-body-temp-path=/var/nginx/client_body_temp --with-http_ssl_module"
Now run the script (pfexec ruby passenger-install-nginx-module), choose customizable installation, give it the source code path, /usr/nginx as the prefix, and no extra configure options – the modification to the script takes care of it. It compiles nginx with gcc and installs it perfectly on top of the OpenSolaris version.
Of course, if you don't want this – there are many reasons not to – alter the extra_args above appropriately and put it in /opt or wherever you like. If you need a new SMF manifest, edit /var/svc/manifest/network/http-nginx.xml appropriately and import it with svccfg import <file> or create a new one with pfexec easy_install Manifold; manifold nginx.xml and import that file.