Pale Moon Sync

For discussions about side projects, potential future projects or helper applications for Pale Moon.
User avatar
Moonchild
Pale Moon guru
Pale Moon guru
Posts: 35402
Joined: 2011-08-28, 17:27
Location: Motala, SE
Contact:

Pale Moon Sync

Unread post by Moonchild » 2014-07-30, 10:38

UPDATE: Sync service taken down as of Sept 29th due to continued issues and disproportionate server load.
Pale Moon 25 will employ the alternative implementation.


Since there are continued problems with the sync service, an "other" project that has been tossed into my lap to take care of because of Mozilla's move to an incompatible "Firefox Accounts" and "Sync 1.5" setup (they should have made it 2.0), a few points in this post to keep you all informed about what's going on:
  1. Pale Moon Sync was set up in accordance with the "run your own sync 1.1 server" documentation available at Mozilla. This involves:
    • A complex python implementation of Weave that does not exactly act like a normal process (service rerstarts fail, for example, and the "worker threads" respond very sluggishly). It also doesn't seem to be very scalable.
    • Using egg/gunicorn as an interface to the front-end, which has already caused issues of its own
    • Passing a whole bunch of things through NGinX to gunicorn, which may be at the root of the authentication issues encountered by people
    This setup seems to be incredibly fragile. It requires frequent restarts and has already caused massive database corruption, as well as simply not working for setting up new accounts, despite everything being set up exactly, to the letter, as instructed.
  2. Additionally, the setup causes load issues when not run on a dedicated server with lots of throughput and iops available.
  3. Because of 1&2, I've been forced to find a different solution. I have found one implemented in PHP, but it's a little tricky to set up. Also, this is not a "full" server implementation yet so some features may be missing as a result, like password recovery options. I'll have to investigate the extent of implementation available.
  4. It looks like (3) can be made to work properly, and I'll use a new server with SSD caching to hopefully avoid (2) altogether, even on a more efficient and less fragile implementation, since there will still be a lot of database reads/writes.
  5. Sync will be down while this is being set up. The current implementation is obviously broken, and I'd rather avoid needless support requests.
  6. Keep in mind that sync is an additional convenience service for Pale Moon provided at no cost to users. I do not have an SLA or even a requirement to keep it available, but wish to do so, anyway, to provide a better experience for Pale Moon users. If maintenance effort or expenses for the sync service get out of hand, I will have to discontinue the service, but I will do my best to keep it available for everyone free of charge.
  7. I may want to ask for help from or hire an experienced PHP programmer to expand on the implementation in PHP (FSyncMS) to add any missing features there may be for account management.
"Sometimes, the best way to get what you want is to be a good person." -- Louis Rossmann
"Seek wisdom, not knowledge. Knowledge is of the past; wisdom is of the future." -- Native American proverb
"Linux makes everything difficult." -- Lyceus Anubite

InS4N3

Re: Pale Moon Sync

Unread post by InS4N3 » 2014-07-30, 11:07

That's sad. Gotta hate Mozilla these days. If it is not this, i't that or whatever.
I hope that a rational solution will be found, I can't see me using a non sync capable web browser =/.
Self sync with a personal device would be at least better than not sync at all.
Best luck about it!

User avatar
Trippynet
Fanatic
Fanatic
Posts: 238
Joined: 2014-06-03, 20:21
Location: UK

Re: Pale Moon Sync

Unread post by Trippynet » 2014-07-30, 14:40

MoonChild wrote:Keep in mind that sync is an additional convenience service for Pale Moon provided at no cost to users. I do not have an SLA or even a requirement to keep it available, but wish to do so, anyway, to provide a better experience for Pale Moon users. If maintenance effort or expenses for the sync service get out of hand, I will discontinue the service.
Fully appreciate that, and thanks for the work that is being done with this.

BoFiS

Re: Pale Moon Sync

Unread post by BoFiS » 2014-07-30, 15:12

Thank you SO MUCH for your continued support and work on this seemingly stupid issue they caused. Any chance that bringing it up to Sync 1.5 and just continuing to rely on Mozilla's sync servers might just be easier? or even possible?

Mikhoul

Re: Pale Moon Sync

Unread post by Mikhoul » 2014-07-30, 16:11

InS4N3 wrote:I can't see me using a non sync capable web browser =/.
Self sync with a personal device would be at least better than not sync at all.
Best luck about it!
Same here I use 3-4 computers that need to be in Sync, for my Bookmarks and numerous add-ons but also for my custom greasemonkey Userscript + Stylish.. All is synced.

In 2014 I can't see a browser with no sync capabilities, worst case why not using the 1.5 SynchServer at least it will work and nobody is is forced to use it if they think it is not enough secure... ;-)

Godd Luck ! :thumbup: :)

User avatar
Moonchild
Pale Moon guru
Pale Moon guru
Posts: 35402
Joined: 2011-08-28, 17:27
Location: Motala, SE
Contact:

Re: Pale Moon Sync

Unread post by Moonchild » 2014-07-30, 16:27

I won't implement Sync 1.5.

It might be the "easy way out" but I have no interest in tying Pale Moon into a service operated by the Mozilla Corporation that is simply 2 steps down (if not more) from Sync 1.1. See other threads on this forum about sync.next.

If that is reason for you to leave Pale Moon behind, then that is entirely up to you, of course.

In the meantime, while I'm sorting things out for the public sync service, if sync is imperative and you can't live a day without it, you can set up your own private sync 1.1 server as well. https://docs.services.mozilla.com/howto ... run-sync11
"Sometimes, the best way to get what you want is to be a good person." -- Louis Rossmann
"Seek wisdom, not knowledge. Knowledge is of the past; wisdom is of the future." -- Native American proverb
"Linux makes everything difficult." -- Lyceus Anubite

Mikhoul

Re: Pale Moon Sync

Unread post by Mikhoul » 2014-07-30, 17:18

Moonchild wrote: In the meantime, while I'm sorting things out for the public sync service, if sync is imperative and you can't live a day without it, you can set up your own private sync 1.1 server as well. https://docs.services.mozilla.com/howto ... run-sync11
Thanks I will take a look and wait one week to see what will happen with PM and synch.

Sorry but I really need it from you or Mozilla I don't really care I'm not paranoid about security, average & common sense is the best words that describe my view on security.

Thanks again and good luck with the PHP implantation I wish the best !!!! :thumbup:

User avatar
Moonchild
Pale Moon guru
Pale Moon guru
Posts: 35402
Joined: 2011-08-28, 17:27
Location: Motala, SE
Contact:

Re: Pale Moon Sync

Unread post by Moonchild » 2014-07-30, 19:13

Mikhoul wrote:I'm not paranoid about security, average & common sense is the best words that describe my view on security.
Then we are on the same page :)
I'm just very aware of how sensitive data is that gets synced, and the new Firefox way IMHO simply doesn't provide strong enough data protection for common sense/average security levels.
"Sometimes, the best way to get what you want is to be a good person." -- Louis Rossmann
"Seek wisdom, not knowledge. Knowledge is of the past; wisdom is of the future." -- Native American proverb
"Linux makes everything difficult." -- Lyceus Anubite

User avatar
Trippynet
Fanatic
Fanatic
Posts: 238
Joined: 2014-06-03, 20:21
Location: UK

Re: Pale Moon Sync

Unread post by Trippynet » 2014-07-30, 21:31

A fair point. There's a big difference between your own data (which you know, understand and take personal responsibility for), and that of other users (which you understandably have to be more careful with).

User avatar
Moonchild
Pale Moon guru
Pale Moon guru
Posts: 35402
Joined: 2011-08-28, 17:27
Location: Motala, SE
Contact:

Re: Pale Moon Sync

Unread post by Moonchild » 2014-08-03, 09:54

OK, an update to this. I'm about ready to give up on alternative solutions, since none of them work.
  • FSyncMS: A PHP implementation of Sync 1.1, properly works as long as you use sync with 1 device. It currently does not work trying to sync with other devices (which kind of defeats the point).
    If someone experienced in PHP can look at why it fails to send collection IDs across, this may be possible to make work.
    https://github.com/MoonchildProductions/FSyncMS
  • OwnCloud has a Firefox sync module, but won't work for a public server (it requires pre-made user accounts).
  • Sync 1.5 server is still incompatible with Sync 1.1 data/clients.
  • Mozilla's "server-full" python-based sync 1.1 server remains extremely unstable despite what I've tried to do to fix it.
Currently there simply isn't a usable solution, and I'm running out of both patience and time trying to get it to work.
If someone can offer to provide a public sync 1.1 server for the Pale Moon project and/or help me with setting up a stable one, that would be awesome.
"Sometimes, the best way to get what you want is to be a good person." -- Louis Rossmann
"Seek wisdom, not knowledge. Knowledge is of the past; wisdom is of the future." -- Native American proverb
"Linux makes everything difficult." -- Lyceus Anubite

User avatar
Moonchild
Pale Moon guru
Pale Moon guru
Posts: 35402
Joined: 2011-08-28, 17:27
Location: Motala, SE
Contact:

Re: Pale Moon Sync

Unread post by Moonchild » 2014-08-03, 20:31

For the time being, the sync service has been enabled again on the relatively unstable but at least working Mozilla implementation of weave. A different version of the weave server is in use now, that will hopefully at least prevent most errors. Previously existing accounts have been cleared - you will have to unlink your device and create a new account again. Sorry for this rather bumpy ride, folks, but I'm trying to provide the best service with what I have.
"Sometimes, the best way to get what you want is to be a good person." -- Louis Rossmann
"Seek wisdom, not knowledge. Knowledge is of the past; wisdom is of the future." -- Native American proverb
"Linux makes everything difficult." -- Lyceus Anubite

User avatar
Antonius32
Add-ons Team
Add-ons Team
Posts: 695
Joined: 2014-05-25, 11:18
Location: Netherlands

Re: Pale Moon Sync

Unread post by Antonius32 » 2014-08-03, 21:40

Thank you for your hard work to get the sync service up and running again; I really appreciate it!

inventif

Re: Pale Moon Sync

Unread post by inventif » 2014-08-03, 22:14

Moonchild,

Is there something the Pale Moon community could do to help out? I mean, besides being patient...

Also, are the last sync accounts completely wiped out or ... ? I tried resetting the password, and it gave me a 404.

User avatar
Moonchild
Pale Moon guru
Pale Moon guru
Posts: 35402
Joined: 2011-08-28, 17:27
Location: Motala, SE
Contact:

Re: Pale Moon Sync

Unread post by Moonchild » 2014-08-03, 23:14

As stated accounts have been cleared, you have to unlink your devices and create a new account.
"Sometimes, the best way to get what you want is to be a good person." -- Louis Rossmann
"Seek wisdom, not knowledge. Knowledge is of the past; wisdom is of the future." -- Native American proverb
"Linux makes everything difficult." -- Lyceus Anubite

User avatar
Moonchild
Pale Moon guru
Pale Moon guru
Posts: 35402
Joined: 2011-08-28, 17:27
Location: Motala, SE
Contact:

Re: Pale Moon Sync

Unread post by Moonchild » 2014-08-03, 23:35

inventif wrote:Is there something the Pale Moon community could do to help out? I mean, besides being patient...
I already asked in one of the previous posts in this very thread: If someone can run a public sync 1.1 server and maintain it, or help me with setting up a working one, myself, that would be a big help.

After switching on the server, I'm once again very quickly seeing degrading service. When just started, it runs fine. When running for a little while, the sync server starts throwing lots of 401s on previously perfectly fine accounts, and 404s on service pages like password resets. Since the python blob is pretty much a mystery to me (also because I don't speak python), I don't have the means to troubleshoot this. Since it's intermittent and increasing in errors, it's not something that is a hard error in the python either, by the looks of it.

I'm running this behind nginx with gunicorn - as explained on the services page of mozilla. It should work, and it initially does, but stuff falls over very quickly.
I don't know what mozilla is running on their servers as a front-end either, or how well-tested the setup with nginx is. It may simply be broken and need apache or lighttpd. But i really don't have the time to do a full trial and error on this right now.

I may simply be forced to build a sync 1.5 client into Pale moon and be done with it, because at this rate, things simply not working, is not a good way to go about it either.
"Sometimes, the best way to get what you want is to be a good person." -- Louis Rossmann
"Seek wisdom, not knowledge. Knowledge is of the past; wisdom is of the future." -- Native American proverb
"Linux makes everything difficult." -- Lyceus Anubite

User avatar
Antonius32
Add-ons Team
Add-ons Team
Posts: 695
Joined: 2014-05-25, 11:18
Location: Netherlands

Re: Pale Moon Sync

Unread post by Antonius32 » 2014-08-04, 09:39

Moonchild wrote:I already asked in one of the previous posts in this very thread: If someone can run a public sync 1.1 server and maintain it, or help me with setting up a working one, myself, that would be a big help.
I'm afraid I can't help you with that, but I hope someone else who reads this, can!
Moonchild wrote:I may simply be forced to build a sync 1.5 client into Pale moon and be done with it, because at this rate, things simply not working, is not a good way to go about it either.
I would regret that, because I much prefer Sync 1.1 over Sync 1.5. I wonder how other Pale Moon users feel about this..

I searched the internet for information on setting up Sync 1.1 on an Apache server, and I found two short (non-Mozilla) tutorials for that. For what it's worth, I'll post the links here:

http://terminal28.com/how-to-install-and-configure-own-firefox-sync-server-weave-debian/

http://avih.github.io/blog/2013/05/31/private-firefox-sync-server-in-5-mins/

User avatar
Moonchild
Pale Moon guru
Pale Moon guru
Posts: 35402
Joined: 2011-08-28, 17:27
Location: Motala, SE
Contact:

Re: Pale Moon Sync

Unread post by Moonchild » 2014-08-04, 12:08

@antonius; thanks for the links.
I've been looking at FSyncMS and it's promising, but something is broken in it which prevents cross-syncing data between devices. IDs come back empty, which prevents the server from serving expected ciphertexts and the sync fails. I'm asking the dev, but don't know how active this server solution is still developed or supported.
"Sometimes, the best way to get what you want is to be a good person." -- Louis Rossmann
"Seek wisdom, not knowledge. Knowledge is of the past; wisdom is of the future." -- Native American proverb
"Linux makes everything difficult." -- Lyceus Anubite

InS4N3

Re: Pale Moon Sync

Unread post by InS4N3 » 2014-08-04, 14:51

I think that, if you have it easier building a 1.5 server in the near future. Go for it, just do what anyone else could do in your position. Let people know in a main thread that this 1.5 has security issues, and that your personal data may be compromised. Let it go like that, I will use it anyway. I don't have that to much sensitive data in my palemoon sync. Everything which matters to me, runs outside a computer most of the time.
Just saying, if 1.5 is as hard or harder than 1.1 then it's more like a game over.

zosodk69

Re: Pale Moon Sync

Unread post by zosodk69 » 2014-08-04, 15:59

Moonchild wrote:I may simply be forced to build a sync 1.5 client into Pale moon and be done with it, because at this rate, things simply not working, is not a good way to go about it either.
This would be unfortunate as there are some of us who are currently running our own sync 1.1 servers. I'm guessing a move to 1.5 would entail removing 1.1 compatibility.

Moonchild, I don't know if it would help but I'm currently running Weave in a docker container. I've got a three container setup consisting of Postgres, Nginx, and Weave. I'm using the mozilla instructions located here: https://docs.services.mozilla.com/howtos/run-sync.html. For the three users and ten connected devices it works pretty well. If the problems you're seeing are scalability related then I doubt I can help but my setup may be useful. If you like I can scrub my dockerfiles and ship them to you.

User avatar
Moonchild
Pale Moon guru
Pale Moon guru
Posts: 35402
Joined: 2011-08-28, 17:27
Location: Motala, SE
Contact:

Re: Pale Moon Sync

Unread post by Moonchild » 2014-08-04, 23:22

I'm making some progress. The problem is scalability and I'm afraid the Mozilla weave solution simply won't work in that respect. I've got it running again as-is now (nginx/MySQL/gunicorn/python-weave) after figuring out that the Mozilla instructions for nginx were not working in terms of authentication. I'll still be looking into a more "native" solution, and am actually making progress with FSyncMS (you can try it out by using a custom server, https://pmsync.palemoon.net/sync/ ) which would be preferable and likely less fragile (since it's nginx/php/mysql). It'll likely still need some extra work to become a more "full" solution.

So it does look like I can avoid Sync 1.5, but time will have to tell.
"Sometimes, the best way to get what you want is to be a good person." -- Louis Rossmann
"Seek wisdom, not knowledge. Knowledge is of the past; wisdom is of the future." -- Native American proverb
"Linux makes everything difficult." -- Lyceus Anubite

Locked