Zimbra Issues
Post upgrade problems
We recently upgraded to Zimbra 8.6 from 8.0.7. We hit a problem that not even zimbra support was able to figure out. Perhaps its that our Zimbra instance has been upgraded since version 3 (or 4) but it got into a bad state whenever we tried to update the proxy. Here's the solution.
First the problem;
$ zmproxyctl status
zmnginxctl is not running
$ zmproxyctl restart
Stopping nginx...nginx is not running.
Starting nginx...nginx: [emerg] invalid port in upstream "mail.example.com:0"
in /opt/zimbra/conf/nginx/includes/nginx.conf.web:62
failed.
So that's fairly easy to find.
Line 62 in conf/nginx/includes/nginx.conf.web
60 upstream zimbra_adminclient
61 {
62 server mail.example.com:0 fail_timeout=60s version=8.6.0_GA_1153;
63 zmauth_admin;
64 }
65 # Define the collection of upstream admin console servers to which we will
66 # proxy. Define each server:port against a server directive
67 #
68 upstream zimbra_admin
69 {
70 server mail.example.com:0 fail_timeout=60s version=8.6.0_GA_1153;
71 zmauth_admin;
72 }
The problem is the :0
Nginx doesn't like that as a port specification.
Looking at the template - in this case nginx.conf.web.template
upstream ${web.admin.upstream.name}
{
${web.admin.upstream.:servers}
zmauth_admin;
}
So what defines that. A google gives me this;
https://wiki.zimbra.com/wiki/Zimbra_Proxy_Manual:Configuration_And_Template_Files_And_Proxy_Related_Variables#web.admin.upstream.:servers
which says;
NGINX Keyword: web.admin.upstream.:servers Description: List of upstream admin servers used by Web Proxy
(i.e. admin servers for which zimbraReverseProxyLookupTarget is true, and whose
mail mode is http|mixed|both) Controlling Attribute: zimbraReverseProxyLookupTarget Default Value: [] Current Value: [<server>:7071] Config Text: server <server>:7071; How to modify: N/A
Not exactly helpful. So we called Zimbra support, who suggested that we
- Run zmproxyconfig (opt/zimbra/libexec/zmproxyconfig -e -o -w -m -C -n 7071:9071 -H `zmhostname`) - which takes down the server, as nginx fails to start.
- Remove all files in conf/nginx/include and re-running zmproxyconfgen - this also did not help
Digging into zmproxyconfgen arguments we find the -D flag.
LDAP configuration (and processing
overrides). -D requires -s upstream server.
If "-s upstream server" is not specified, it
just dumps the default varaible map
So, diving in and looking for our problem area;
libexec/zmproxyconfgen -D -s mail.example.com
NGINX Keyword: web.admin.upstream.:servers
Description: List of upstream HTTPS Admin client servers used by Web Proxy
Value Type: CUSTOM
Controlling Attribute: (none)
Default Value: (none)
Current Value: [mail.example.com:0 fail_timeout=60s version=8.6.0_GA_1153]
Config Text: server mail.example.com:0 fail_timeout=60s version=8.6.0_GA_1153;
So there's our :0 again.
And elsewhere in the same output (because we know it should be 7071) we find;
NGINX Keyword: web.admin.uport
Description: Admin console upstream port
Value Type: INTEGER
Controlling Attribute: zimbraAdminPort
Default Value: 7071
Current Value: 7071
Config Text: 7071
We can look for zimbraAdminPort in the output and compare it to a fresh install on a test server;
1) A working test server;
$ zmprov gacf | grep AdminPort
zimbraAdminPort: 7071
zimbraReverseProxyAdminPortAttribute: zimbraAdminPort
2) Our problem server
$ zmprov -l gacf | grep AdminPort
zimbraAdminPort: 7071
In other words, we're missing the zimbraReverseProxyAdminPortAttribute.
Are there any other attributes we're missing?
On the clean install;
$ zmprov gacf | grep Attribute
zimbraContactHiddenAttributes: dn,vcardUID,vcardURL,vcardXProps,member
zimbraReverseProxyAdminPortAttribute: zimbraAdminPort
zimbraReverseProxyDomainNameAttribute: zimbraDomainName
zimbraReverseProxyHttpPortAttribute: zimbraMailPort
zimbraReverseProxyHttpSSLPortAttribute: zimbraMailSSLPort
zimbraReverseProxyImapPortAttribute: zimbraImapBindPort
zimbraReverseProxyImapSSLPortAttribute: zimbraImapSSLBindPort
zimbraReverseProxyMailHostAttribute: zimbraMailHost
zimbraReverseProxyPop3PortAttribute: zimbraPop3BindPort
zimbraReverseProxyPop3SSLPortAttribute: zimbraPop3SSLBindPort
On the problem server
[zimbra@co1mail01 ~]$ zmprov gacf | grep Attribute
zimbraContactHiddenAttributes: dn,vcardUID,vcardURL,vcardXProps,member
zimbraReverseProxyDomainNameAttribute: zimbraDomainName
zimbraReverseProxyHttpPortAttribute: zimbraMailPort
zimbraReverseProxyHttpSSLPortAttribute: zimbraMailSSLPort
zimbraReverseProxyImapPortAttribute: zimbraImapBindPort
zimbraReverseProxyImapSSLPortAttribute: zimbraImapSSLBindPort
zimbraReverseProxyMailHostAttribute: zimbraMailHost
zimbraReverseProxyPop3PortAttribute: zimbraPop3BindPort
zimbraReverseProxyPop3SSLPortAttribute: zimbraPop3SSLBindPort
So, it looks like its just that one.
Fixing it using;
$ zmprov -l mcf zimbraReverseProxyAdminPortAttribute zimbraAdminPort
$ zmprov -l gacf zimbraReverseProxyAdminPortAttribute
zimbraReverseProxyAdminPortAttribute: zimbraAdminPort
Regenerating config and checking;
$ libexec/zmproxyconfgen -D -s mail.example.com > /tmp/zmproxyconfgen.out2
$ diff -u /tmp/zmproxyconfgen.out /tmp/zmproxyconfgen.out2
--- /tmp/zmproxyconfgen.out 2016-02-09 12:30:42.856618559 +0000
+++ /tmp/zmproxyconfgen.out2 2016-02-09 12:42:37.146391380 +0000
@@ -607,8 +607,8 @@
Value Type: CUSTOM
Controlling Attribute: (none)
Default Value: (none)
- Current Value: [mail.example.com:0 fail_timeout=60s version=8.6.0_GA_1153]
- Config Text: server mail.example.com:0 fail_timeout=60s version=8.6.0_GA_1153;
+ Current Value: [mail.example.com:7071 fail_timeout=60s version=8.6.0_GA_1153]
+ Config Text: server mail.example.com:7071 fail_timeout=60s version=8.6.0_GA_1153;
NGINX Keyword: web.admin.upstream.:servers
@@ -616,8 +616,8 @@
Value Type: CUSTOM
Controlling Attribute: (none)
Default Value: (none)
- Current Value: [mail.example.com:0 fail_timeout=60s version=8.6.0_GA_1153]
- Config Text: server mail.example.com:0 fail_timeout=60s version=8.6.0_GA_1153;
+ Current Value: [mail.example.com:7071 fail_timeout=60s version=8.6.0_GA_1153]
+ Config Text: server mail.example.com:7071 fail_timeout=60s version=8.6.0_GA_1153;
now, looking again in conf/nginx/includes
upstream zimbra_adminclient
{
server mail.example.com:7071 fail_timeout=60s version=8.6.0_GA_1153;
zmauth_admin;
}
# Define the collection of upstream admin console servers to which we will
# proxy. Define each server:port against a server directive
#
upstream zimbra_admin
{
server mail.example.com:7071 fail_timeout=60s version=8.6.0_GA_1153;
zmauth_admin;
}
Problem fixed. nginx now starts.
All we need to do now is figure out why imap completely fails when we have the proxy active.
Hope that helped.
