Set up Pale Moon sync server (Mozilla/Python)

Post your tutorials for using applications or performing related tasks here.
Note: Not for "how do I...?" Questions!
Forum rules
Tutorials and Howtos should only relate to developed software, and not to third party applications. e.g.: Don't post a generic Howto for configuring a firewall.
If you have a question how to do something, you should use one of the support boards, not this board. It is meant for people to document and post instructions.
go4pale

Set up Pale Moon sync server (Mozilla/Python)

Unread post by go4pale » 2017-01-17, 10:54

Hello Pale Moon users

first of all: thanks to all that helps/helped to develop the Pale Moon browser.

so! next you will find what I did to run my own sync server for Pale Moon 27.0.3 (language german).

the system I run the sync server is a debian 8 with a minimal xfce4 desktop. for this I used a debian 8 net install image and then I installed a minimal desktop. I also set up sudo and disabled su.
a) minimal desktop:

Code: Select all

apt-get install xorg lightdm lightdm-gtk-greeter xfwm4 xfdesktop xfconf xfce4-settings xfce4-session xfce4-panel xfce4-terminal libxfce4ui-utils gtk2-engines-xfce thunar thunar-volman thunar-archive-plugin gksu sudo build-essential firefox-esr geany geany-plugins synaptic xdg-user-dirs gtk-engines-murrine murrine-themes
set up sudo:

Code: Select all

usermod -a -G sudo username
visudo (adapt the sudo line, so that I haven't to use password)

Code: Select all

reboot
login to the minimal desktop an become root:

Code: Select all

sudo -s
b) set up an apache mysql php5 server

Code: Select all

apt-get install mariadb-server mariadb-client (tip: write down the password, so one can forget it)
apt-get install apache2
apt-get install php5 libapache2-mod-php5
apt-get install php5-mysqlnd php5-curl php5-gd php5-intl php-pear php5-imagick php5-imap php5-mcrypt php5-memcache php5-pspell php5-recode php5-snmp php5-sqlite php5-tidy php5-xmlrpc php5-xsl
apt-get install php5-apcu
apt-get install phpmyadmin
comment:
further I change the settings of apache php5 and mysql to use utf-8 (search in the internet how to do that).

c) next I installed packages needed for the sync server:

Code: Select all

apt-get install libssl-dev python-mysqldb python-mysql.connector python-dev mercurial python-virtualenv libapache2-mod-wsgi libmysqlclient-dev python make
d) create a sync server owner (in order to run the sync server and set the permissions) and group

Code: Select all

useradd -d /usr/local/fsync -m -r -U -s /bin/false fsync
e) add user www-data to the new group fsync

Code: Select all

usermod -a -G fsync www-data
f) go to the fsync home an install the mozilla sync server version 1.1

Code: Select all

cd /usr/local/fsync
hg clone https://hg.mozilla.org/services/server-full (creates a folder server-full within the sync server software)
cd server-full
make build
g) add some folders within the folder fsync

Code: Select all

cd /usr/local/fsync
mkdir -p tmp
mkdir -p logs
h) set permissons below fsync folder (have you set up acl?)

Code: Select all

cd /usr/local
chown -R fsync.fsync fsync
find /usr/local/fsync -type d -exec chmod 2770 "{}" \; -exec setfacl -m u::rwX,g::rwX,o::--- "{}" \;
find /usr/local/fsync -type f -executable -exec chmod 770 "{}" \; -exec setfacl -m u::rwx,g::rwx,o::--- "{}" \;
find /usr/local/fsync -type f ! -executable -exec chmod 660 "{}" \; -exec setfacl -m u::rw-,g::rw-,o::--- "{}" \;
i) set up a mysql database (I normaly to that using phpmyadmin) below is a command line version I found in the net (one have to log into the database as root)

Code: Select all

create database fsync;
GRANT ALL PRIVILEGES ON fsync .* TO fsync@localhost IDENTIFIED BY 'PASSWORD';
flush privileges;
quit
j) adapt sync server config files

Code: Select all

cd /usr/local/fsync/server-full
cp -p etc/sync.conf etc/sync.conf-orig
rm etc/sync.conf
cp -p etc/mysql.conf etc/sync.conf
cp -p development.ini development.ini-orig
cp -p sync.wsgi sync.wsgi-orig
nano etc/sync.conf

Code: Select all

[captcha]
...
use_ssl = true
...

[storage]
...
sqluri = mysql://fsync:PASSWORD@localhost:3306/fsync (hint: PASSWORD one used when the database was created) 
...
quota_size = 10240
...

[auth]
...
sqluri = mysql://fsync:PASSWORD@localhost:3306/fsync

[nodes]
fallback_node = https://fsync.example.com (hint: domain one had set up using dns provider)
...

[reset_codes]
backend = services.resetcodes.rc_sql.ResetCodeSQL
sqluri = mysql://fsync:PASSWORD@localhost:3306/fsync
create_tables = True

[keyexchange]
use_memory = true
nano development.ini

Code: Select all

[DEFAULT]
debug = False
...

[handler_syncserver_errors]
...
args = ('/usr/local/fsync/logs/sync-error.log',)
...
nano sync.wsgi

Code: Select all

...
# setting up the egg cache to a place where apache can write
os.environ['PYTHON_EGG_CACHE'] = '/usr/local/fsync/tmp/python-eggs'
...

k) create virtaul apache2 host. in order to use https one needs a trusted certificate (for example comodo); I managed to use self signed certificates; in this case you have first to accept the self signed certificate in palemoon. start palemoon, use the url https://fsync.example.com, accept the self signed certificate, then go to the sync settings of palemoon.

create a self signed certificate:

Code: Select all

cd /etc/apache
mkdir -p ssl/fsync
cd /etc/apache2/ssl/fsync
make-ssl-cert /usr/share/ssl-cert/ssleay.cnf /etc/apache2/ssl/fsync/fsync.example.com.cert (add the your domain, when asked for)
touch fsync.example.com.key
touch fsync.example.com.crt
cat fsync.example.com.cert (copy the key)
nano  fsync.example.com.key (past in the key)
cat fsync.example.com.cert (copy the certificate)
nano  fsync.example.com.crt (paste in the certificate)
modify the apache default http/https virtual hosts (hint: I'm far away from being an apache expert; but it's the configuration I use for my running sync server)

Code: Select all

cd /etc/apache2/sites-available
nano 000-default.conf

Code: Select all

<VirtualHost *:80>
	# The ServerName directive sets the request scheme, hostname and port that
	# the server uses to identify itself. This is used when creating
	# redirection URLs. In the context of virtual hosts, the ServerName
	# specifies what hostname must appear in the request's Host: header to
	# match this virtual host. For the default virtual host (this file) this
	# value is not decisive as it is used as a last resort host regardless.
	# However, you must set it for any further virtual host explicitly.
	#ServerName http://www.example.com

	ServerAdmin webmaster@localhost
	DocumentRoot /var/www/html

	# Available loglevels: trace8, ..., trace1, debug, info, notice, warn,
	# error, crit, alert, emerg.
	# It is also possible to configure the loglevel for particular
	# modules, e.g.
	#LogLevel info ssl:warn

	ErrorLog ${APACHE_LOG_DIR}/error.log
	CustomLog ${APACHE_LOG_DIR}/access.log combined

	# For most configuration files from conf-available/, which are
	# enabled or disabled at a global level, it is possible to
	# include a line for only one particular virtual host. For example the
	# following line enables the CGI configuration for this host only
	# after it has been globally disabled with "a2disconf".
	#Include conf-available/serve-cgi-bin.conf

	# the rewrite settings are added to the default debian virtual host 000-default.conf
	# redirect www to non-www
	RewriteEngine On
	RewriteCond %{HTTP_HOST} !^$
	RewriteCond %{HTTP_HOST} ^www\.(.+) [NC]
	RewriteRule ^/(.*) http://%1/$1 [L,R=301]
    
	# redirect http requests to https
	RewriteEngine On
	RewriteCond %{HTTP_HOST} !^localhost	
	RewriteCond %{HTTPS} off
	RewriteRule (.*) https://%{HTTP_HOST}:443%{REQUEST_URI} [R=301,L][/color]

</VirtualHost>

Code: Select all

cp default-ssl.conf fsync.example.com-ssl.conf
nano fsync.example.com-ssl.conf

Code: Select all

<IfModule mod_ssl.c>
	<VirtualHost *:443>
		ServerName fsync.example.com

		ServerAdmin webmaster@localhost

		DocumentRoot /usr/local/fsync/server-full

		# Available loglevels: trace8, ..., trace1, debug, info, notice, warn,
		# error, crit, alert, emerg.
		# It is also possible to configure the loglevel for particular
		# modules, e.g.
		#LogLevel info ssl:warn

		ErrorLog /usr/local/fsync/logs/error.log
		CustomLog /usr/local/fsync/logs/access.log combined

		# For most configuration files from conf-available/, which are
		# enabled or disabled at a global level, it is possible to
		# include a line for only one particular virtual host. For example the
		# following line enables the CGI configuration for this host only
		# after it has been globally disabled with "a2disconf".
		#Include conf-available/serve-cgi-bin.conf

		#   SSL Engine Switch:
		#   Enable/Disable SSL for this virtual host.
		SSLEngine on

		#   A self-signed (snakeoil) certificate can be created by installing
		#   the ssl-cert package. See
		#   /usr/share/doc/apache2/README.Debian.gz for more info.
		#   If both key and certificate are stored in the same file, only the
		#   SSLCertificateFile directive is needed.
		SSLCertificateFile	/etc/apache2/ssl/fsync/fsync.example.com.crt
		SSLCertificateKeyFile /etc/apache2/ssl/fsync/fsync.example.com.key

		#   Server Certificate Chain:
		#   Point SSLCertificateChainFile at a file containing the
		#   concatenation of PEM encoded CA certificates which form the
		#   certificate chain for the server certificate. Alternatively
		#   the referenced file can be the same as SSLCertificateFile
		#   when the CA certificates are directly appended to the server
		#   certificate for convinience.
		#SSLCertificateChainFile /etc/apache2/ssl.crt/server-ca.crt

		#   Certificate Authority (CA):
		#   Set the CA certificate verification path where to find CA
		#   certificates for client authentication or alternatively one
		#   huge file containing all of them (file must be PEM encoded)
		#   Note: Inside SSLCACertificatePath you need hash symlinks
		#		 to point to the certificate files. Use the provided
		#		 Makefile to update the hash symlinks after changes.
		#SSLCACertificatePath /etc/ssl/certs/
		#SSLCACertificateFile /etc/apache2/ssl.crt/ca-bundle.crt

		#   Certificate Revocation Lists (CRL):
		#   Set the CA revocation path where to find CA CRLs for client
		#   authentication or alternatively one huge file containing all
		#   of them (file must be PEM encoded)
		#   Note: Inside SSLCARevocationPath you need hash symlinks
		#		 to point to the certificate files. Use the provided
		#		 Makefile to update the hash symlinks after changes.
		#SSLCARevocationPath /etc/apache2/ssl.crl/
		#SSLCARevocationFile /etc/apache2/ssl.crl/ca-bundle.crl

		#   Client Authentication (Type):
		#   Client certificate verification type and depth.  Types are
		#   none, optional, require and optional_no_ca.  Depth is a
		#   number which specifies how deeply to verify the certificate
		#   issuer chain before deciding the certificate is not valid.
		#SSLVerifyClient require
		#SSLVerifyDepth  10

		#   SSL Engine Options:
		#   Set various options for the SSL engine.
		#   o FakeBasicAuth:
		#	 Translate the client X.509 into a Basic Authorisation.  This means that
		#	 the standard Auth/DBMAuth methods can be used for access control.  The
		#	 user name is the `one line' version of the client's X.509 certificate.
		#	 Note that no password is obtained from the user. Every entry in the user
		#	 file needs this password: `xxj31ZMTZzkVA'.
		#   o ExportCertData:
		#	 This exports two additional environment variables: SSL_CLIENT_CERT and
		#	 SSL_SERVER_CERT. These contain the PEM-encoded certificates of the
		#	 server (always existing) and the client (only existing when client
		#	 authentication is used). This can be used to import the certificates
		#	 into CGI scripts.
		#   o StdEnvVars:
		#	 This exports the standard SSL/TLS related `SSL_*' environment variables.
		#	 Per default this exportation is switched off for performance reasons,
		#	 because the extraction step is an expensive operation and is usually
		#	 useless for serving static content. So one usually enables the
		#	 exportation for CGI and SSI requests only.
		#   o OptRenegotiate:
		#	 This enables optimized SSL connection renegotiation handling when SSL
		#	 directives are used in per-directory context.
		#SSLOptions +FakeBasicAuth +ExportCertData +StrictRequire
		<FilesMatch "\.(cgi|shtml|phtml|php)$">
				SSLOptions +StdEnvVars
		</FilesMatch>
		<Directory /usr/lib/cgi-bin>
				SSLOptions +StdEnvVars
 		</Directory>

		#   SSL Protocol Adjustments:
		#   The safe and default but still SSL/TLS standard compliant shutdown
		#   approach is that mod_ssl sends the close notify alert but doesn't wait for
		#   the close notify alert from client. When you need a different shutdown
		#   approach you can use one of the following variables:
		#   o ssl-unclean-shutdown:
		#	 This forces an unclean shutdown when the connection is closed, i.e. no
		#	 SSL close notify alert is send or allowed to received.  This violates
		#	 the SSL/TLS standard but is needed for some brain-dead browsers. Use
		#	 this when you receive I/O errors because of the standard approach where
		#	 mod_ssl sends the close notify alert.
		#   o ssl-accurate-shutdown:
		#	 This forces an accurate shutdown when the connection is closed, i.e. a
		#	 SSL close notify alert is send and mod_ssl waits for the close notify
		#	 alert of the client. This is 100% SSL/TLS standard compliant, but in
		#	 practice often causes hanging connections with brain-dead browsers. Use
		#	 this only for browsers where you know that their SSL implementation
		#	 works correctly.
		#   Notice: Most problems of broken clients are also related to the HTTP
		#   keep-alive facility, so you usually additionally want to disable
		#   keep-alive for those clients, too. Use variable "nokeepalive" for this.
		#   Similarly, one has to force some clients to use HTTP/1.0 to workaround
		#   their broken HTTP/1.1 implementation. Use variables "downgrade-1.0" and
		#   "force-response-1.0" for this.
		BrowserMatch "MSIE [2-6]" \
				nokeepalive ssl-unclean-shutdown \
				downgrade-1.0 force-response-1.0
		# MSIE 7 and newer should be able to use keepalive
		BrowserMatch "MSIE [17-9]" ssl-unclean-shutdown

		# added for the sync server
		WSGIProcessGroup fsync
		WSGIDaemonProcess fsync user=fsync group=fsync processes=2 threads=25 python-path=/usr/local/fsync/server-full/local/lib/python2.7/site-packages
		WSGIPassAuthorization On
		WSGIScriptAlias / /usr/local/fsync/server-full/sync.wsgi

		# added for the sync server
		<Directory /opt/weave>
			Require all granted
		</Directory>

	</VirtualHost>
</IfModule>

# vim: syntax=apache ts=4 sw=4 sts=4 sr noet
l) enable apache modules and the https virtual host restart apache

Code: Select all

cd /etc/apache2/sites-available
a2enmod ssl
a2enmod rewrite
a2enmod wsgi
a2ensite fsync.example.com-ssl.conf
service apache2 restart

m) start the sync server as user fsync

Code: Select all

cd /usr/local/fsync/server-full
sudo -u fsync bin/paster serve development.ini &
n) if you run a test environment, for example in virtualbox modify the /etc/hosts file from virtual client pc and add the ip (ifconfig -a) from the virtual client pc and the test domain. for example

nano /etc/hosts

Code: Select all

...
192.168.56.120  fsync.example.com
o) test your sync server (install palemoon 27.0.3). open palemoon 27.0.3
URL:

Code: Select all

https://fsync.example.com
you should see an error: 404 Not Found
remember: accept the self signed certificate!!!

URL:

Code: Select all

https://fsync.example.com/__heartbeat__
you should see a blank page

URL:

Code: Select all

https://fsync.example.com/weave-password-reset
you should see a mozilla weave page, whre you can reset the password

URL:

Code: Select all

https://fsync.example.com/weave-delete-account
you should see a mozilla weave page, where you can delete an account


p) final: if one wouldn't that forgein users can use your sync server set in the sync.conf

Code: Select all

cd /usr/local/fsync/server-full
nano etc/sync.conf

Code: Select all

...
allow_new_users = false
...
Hope it will work.

Kind Regards, go4pale
Last edited by Moonchild on 2017-01-17, 12:45, edited 1 time in total.

go4pale

Re: Set up Pale Moon sync server (Mozilla/Python)

Unread post by go4pale » 2017-01-17, 14:03

there is a wrong configuration in the fsync.example.com-ssl.conf

the part

Code: Select all

      # added for the sync server
      <Directory /opt/weave>
         Require all granted
      </Directory>
should be

Code: Select all

      # added for the sync server
      <Directory /usr/local/fsync>
         Require all granted
      </Directory>
sorry for this error.

kind regards, go4pale

New Tobin Paradigm

Re: Set up Pale Moon sync server (Mozilla/Python)

Unread post by New Tobin Paradigm » 2017-01-17, 18:35

Why not simply use our fork of FSyncMS https://github.com/MoonchildProductions/FSyncMS. Uses PHP and mysql and isn't a huge mess.

go4pale

Re: Set up Pale Moon sync server (Mozilla/Python)

Unread post by go4pale » 2017-01-18, 01:32

As one can read on FSyncMS
Although the original author has planned further extesnions to this implementation, the current state of this server implementation is rather stagnant ans missing two important features:

Delete account from the web
Reset password from the web (similar to reset inside the client)
So, using Mozillas sync server version 1.1 one has both of these features: delete an account and reset passwords.

By the way: I use your FSyncMS as a "private single user" sync server. Where I am the only user and never use the option of "deleting" or "reseting".

Kind regards, go4pale.

New Tobin Paradigm

Re: Set up Pale Moon sync server (Mozilla/Python)

Unread post by New Tobin Paradigm » 2017-01-18, 08:57

One of these years I will see if I can help out with FSyncMS and make those functions a thing. I really am too busy at the moment.

Locked