Ubuntu07 Aug 2013 01:56 pm

For some time we’ve wanted to phase, gradually roll out, updates to expanding subsets of Ubuntu users so that we may monitor for regressions and stop the update process if there are any. The support for phased updates has existed in update-manager for a while, but we did not have the server side part implemented. Thanks to the work of Colin Watson, Evan Dandrea, and myself this is now done.

Who is participating?

Users of Ubuntu 13.04 who install updates with update-manager are automatically participating in this process. For every package, in -updates, update-manager will generate a random number and if that number is less than the Phased-Update-Percentage the package will be installed. One can opt out of the Phased Update process by adding ‘Update-Manager::Never-Include-Phased-Updates “True”;’ to the configuration file “/etc/apt/apt.conf”.

How does the Phased Update process work?

When a Stable Release Update is released to 13.04 it will have its phased update percentage initially set to 10%. A job is run, every 6 hours, in the data center that checks to see if there are any regression about the package, and if there are none then the phased update percentage will be incremented by 10%. The phased update percentage for a binary package is available at the publishing history page for it. Here is an example with apport. If there is no value for “Phased updates” then the update is fully phased at 100%.

What are the regression checks?

The Ubuntu Error Tracker (errors.ubuntu.com) has been modified to help us determine if there are any regressions about the package. We do this by checking to see if there are any crashes reported about the new version of the package that were not reported about the previous version of the package. (You can actually check this yourself using a query like: https://errors.ubuntu.com/api/1.0/package-version-new-buckets/?format=json&package=unattended-upgrades&previous_version=0.76&new_version=0.76ubuntu1) Additionally, we check the error tracker to see if there is an increased rate of crashes about the package. This is done by examining the quantity of errors reported today and comparing it to the average number of crashes per day for the past two weeks multiplied by the portion of the day that has passed.

If either of these types of regressions are detected then the phasing of the update is stopped by setting it to 0. This will prevent other users from receiving the updated version of the package. There is also a report of packages currently undergoing phasing that displays the phased update percentage for the package and any detected regressions. Additionally, an email is sent to the signer of the package (uploader) and its creator (uploader or sponsee). The email notifies them of the problem and that phasing of the update has been stopped.

There is support in the phased-updater for overriding specific problems, for example if we determine that a regression was not introduced in a specific version of a package. It also keeps track of emails sent so that we do not send an email about the same problem more than once.

If you encounter any issues as a user installing updates or as a developer who has uploaded packages please let me know.

18 Responses to “Phasing of Stable Release Updates”

  1. on 08 Aug 2013 at 5:32 pm N. Friedman

    Are you talking about a system update or the update of ordinary programs? And, if a system update, is this intended to solve the problem of the short support cycle for non-LTS versions of Ubuntu?

    I use 12.04, which, if I understand your post, is not part of your project. Were it the case that there would be system updates that extend the support for a newer system beyond 9 months, that would interest me. Otherwise, I have not, so far as I can recall, ever had a regression using 12.04.

  2. on 09 Aug 2013 at 7:37 am Brian Murray

    I am talking about the update of any package installed on an Ubuntu system which exists in the Ubuntu archive. This will not extend the support cycle of non-LTS versions of Ubuntu.

    I’m glad to hear you have not encountered any regressions using 12.04.

  3. on 09 Aug 2013 at 9:29 am sdb

    Where does update manager get the current Phased-Update-Percentage ?

    It sounds as if I update via apt-get then I will get all the updates immediately. Correct?

    It would be nice if there was a apt conf setting: Update-Manager::Minimum-Phase-Percentage {##}; (or whatever name) where I would receive updates only after the Phased-Update-Percentage was equal to or more than the number I specify.

    I like the phased update idea. I have had numerous regressions with 12.04. (On multiple slightly different Sandy Bridge systems, problems experienced with Intel video and embedded Realtek RTL8111/8168 ethernet).

  4. on 09 Aug 2013 at 9:44 am Brian Murray

    update-manager reads the Phased-Update-Percentage from the packages list at archive.ubuntu.com or your local mirror. Yes, if you use apt-get then you will get all updates immediately.

    Given that phased updates will roll out rather quickly, less than 3 days as currently configured, why would you want to specify some percentage?

  5. on 11 Aug 2013 at 5:41 am jarfil

    So, let me get this straight:

    apt-get users = all updates (semi-stable?)
    Phased packages = random sample (semi-stable to stable)
    packages without “Phased” keyword = actually stable

    Is that it?

    I understand this is intended for mainstream users, but I think some of us would like to have a chance to decide whether we want or not phased packages from apt-get.

  6. on 11 Aug 2013 at 6:57 am Clayton Casciato

    @Brian Murray: If he specifies 51%, then he can have 50% of other users basically testing it “for him” before it potentially screws up his system. If problems are encountered and the packages are rolled back, he won’t be affected because, hopefully, the percentage didn’t get high enough before bug reports rolled in. Basically, @sdb wants to create a buffer between himself and the new updates, using the Ubuntu community as a huge testing base. If there are problems, he has a lower chance of experiencing them because he has 50% of the other users to test it out before him.

  7. on 11 Aug 2013 at 11:21 am petter

    Sounds like a very neat feature! Can you give any specifik examples when the “phasing break” has been activated?

  8. on 11 Aug 2013 at 9:16 pm afunix

    Hello.
    Interesting feature, but what about server editions which doesn’t have update-manager? Are there any plans to patch unattended-upgrades?
    Also there are lots of bugs in Ubuntu which are not fixed for ages due to lack of developer’s interest or slow upstream reaction, etc. It seems that my Ubuntu won’t be able to upgrade some packages until new release becomes available. Is there any kind of notification?

  9. on 12 Aug 2013 at 2:07 am Leon Miklosik

    I would, like sdb, be able to choose. Will I be at the beginning, or at the end of the update.

  10. on 12 Aug 2013 at 4:28 am vrm

    Does this also apply to kubuntu ?

  11. on 12 Aug 2013 at 10:39 am Brian Murray

    @Clayton - I understand why one might want to specify a percentage, but it seems to me that if one wants a buffer they should just wait until the update is fully phased.

  12. on 12 Aug 2013 at 10:42 am Brian Murray

    @afunix - there is a notification as the uploader of the package is sent an email regarding the regression or regressions encountered. Additionally, the uploader of the package has a vested interest in fixing the regression as they uploaded an update to a Stable Release for some reason.

  13. on 12 Aug 2013 at 10:44 am Brian Murray

    @vrm - I’m not aware of how much of the update-manager code base KDE uses. It’d be best to check with a KDE developer.

  14. on 13 Aug 2013 at 1:39 am Keba

    Why havenā€˜t you implemented this feature in apt, so all update tools (apt-get, update-manager and many more) are able to use Phased Upgrades?

  15. on 13 Aug 2013 at 9:51 am wrzomar

    Hi Brian,

    I’ve 2 questions:
    1. What if the package would be updated but its dependencies wouldn’t?
    2. It takes 54 hours without error reports to achieve threshold of 100%. Wouldn’t be better to start from 6.25% (1/16) and double it every 6 hours? It’d occupy 24 hours to achieve 100% and the whole process would last twice shorter.

    Best Regards,
    Mark.

  16. on 14 Aug 2013 at 8:27 am Brian Murray

    @wrzomar

    1) The phased-update-percentage check just indicates that a package is available for installation, the package install resolver still behaves the same way. So if the dependencies were not installable then the package would not be installed.

    2) We are still adjusting the regression checks, after we are comfortable with how the system is behaving we’ll look at adjusting the percentages and time between runs.

  17. on 25 Aug 2013 at 10:35 pm Luke

    Is it possible that these bugs are being caused by bad phased upgrades not reverting properly?

    https://bugs.launchpad.net/unity/+bug/1209442

  18. on 26 Aug 2013 at 11:57 am Brian Murray

    @Luke - thanks for bringing this up, but the answer is no and I’ve updated the bug with details.

Feed on comments to this Post

Leave a Reply