From 4396bc33e8463a58b00bc15394bff8f192fed718 Mon Sep 17 00:00:00 2001 From: Andi Chang Date: Mon, 4 May 2026 12:52:02 -0700 Subject: [PATCH] Add blogger posts --- docs/news/posts/blogger-post-2006-09-23-1.md | 6 + docs/news/posts/blogger-post-2006-09-25-1.md | 6 + docs/news/posts/blogger-post-2006-09-26-1.md | 6 + docs/news/posts/blogger-post-2006-09-29-1.md | 6 + docs/news/posts/blogger-post-2006-10-02-1.md | 6 + docs/news/posts/blogger-post-2006-10-06-1.md | 6 + docs/news/posts/blogger-post-2006-10-12-1.md | 6 + docs/news/posts/blogger-post-2006-10-12-2.md | 6 + docs/news/posts/blogger-post-2006-10-12-3.md | 6 + docs/news/posts/blogger-post-2006-10-13-1.md | 6 + docs/news/posts/blogger-post-2006-10-13-2.md | 6 + docs/news/posts/blogger-post-2006-10-14-1.md | 6 + docs/news/posts/blogger-post-2006-10-15-1.md | 6 + docs/news/posts/blogger-post-2006-10-16-1.md | 6 + docs/news/posts/blogger-post-2006-10-16-2.md | 6 + docs/news/posts/blogger-post-2006-10-16-3.md | 6 + docs/news/posts/blogger-post-2006-10-17-1.md | 6 + docs/news/posts/blogger-post-2006-10-17-2.md | 6 + docs/news/posts/blogger-post-2006-10-18-1.md | 6 + docs/news/posts/blogger-post-2006-10-19-1.md | 6 + docs/news/posts/blogger-post-2006-10-22-1.md | 6 + docs/news/posts/blogger-post-2006-10-26-1.md | 6 + docs/news/posts/blogger-post-2006-10-28-1.md | 6 + docs/news/posts/blogger-post-2006-10-29-1.md | 6 + docs/news/posts/blogger-post-2006-11-05-1.md | 6 + docs/news/posts/blogger-post-2006-11-07-1.md | 6 + docs/news/posts/blogger-post-2006-11-11-1.md | 6 + docs/news/posts/blogger-post-2006-11-11-2.md | 6 + docs/news/posts/blogger-post-2006-11-15-1.md | 6 + docs/news/posts/blogger-post-2006-11-16-1.md | 6 + docs/news/posts/blogger-post-2006-11-20-1.md | 6 + docs/news/posts/blogger-post-2006-11-23-1.md | 6 + docs/news/posts/blogger-post-2006-11-29-1.md | 6 + docs/news/posts/blogger-post-2006-11-30-1.md | 6 + docs/news/posts/blogger-post-2006-12-10-1.md | 6 + docs/news/posts/blogger-post-2007-02-04-1.md | 6 + docs/news/posts/blogger-post-2007-02-04-2.md | 6 + docs/news/posts/blogger-post-2007-02-05-1.md | 6 + docs/news/posts/blogger-post-2007-02-26-1.md | 6 + docs/news/posts/blogger-post-2007-03-04-1.md | 6 + docs/news/posts/blogger-post-2007-03-06-1.md | 6 + docs/news/posts/blogger-post-2007-03-11-1.md | 6 + docs/news/posts/blogger-post-2007-03-24-1.md | 6 + docs/news/posts/blogger-post-2007-03-25-1.md | 6 + docs/news/posts/blogger-post-2007-03-29-1.md | 6 + docs/news/posts/blogger-post-2007-05-02-1.md | 6 + docs/news/posts/blogger-post-2007-05-20-1.md | 6 + docs/news/posts/blogger-post-2007-06-01-1.md | 6 + docs/news/posts/blogger-post-2007-06-15-1.md | 6 + docs/news/posts/blogger-post-2007-06-19-1.md | 6 + docs/news/posts/blogger-post-2007-06-30-1.md | 6 + docs/news/posts/blogger-post-2007-07-02-1.md | 6 + docs/news/posts/blogger-post-2007-08-01-1.md | 6 + docs/news/posts/blogger-post-2007-08-04-1.md | 6 + docs/news/posts/blogger-post-2007-09-07-1.md | 6 + docs/news/posts/blogger-post-2007-09-22-1.md | 6 + docs/news/posts/blogger-post-2007-10-08-1.md | 6 + docs/news/posts/blogger-post-2007-10-08-2.md | 6 + docs/news/posts/blogger-post-2007-10-18-1.md | 6 + docs/news/posts/blogger-post-2007-10-18-2.md | 6 + docs/news/posts/blogger-post-2007-11-03-1.md | 6 + docs/news/posts/blogger-post-2008-01-31-1.md | 6 + docs/news/posts/blogger-post-2008-02-29-1.md | 6 + docs/news/posts/blogger-post-2008-03-12-1.md | 6 + docs/news/posts/blogger-post-2008-03-15-1.md | 6 + docs/news/posts/blogger-post-2008-03-18-1.md | 6 + docs/news/posts/blogger-post-2008-03-21-1.md | 6 + docs/news/posts/blogger-post-2008-03-25-1.md | 6 + docs/news/posts/blogger-post-2008-04-18-1.md | 6 + docs/news/posts/blogger-post-2008-05-17-1.md | 6 + docs/news/posts/blogger-post-2008-05-20-1.md | 6 + docs/news/posts/blogger-post-2008-05-23-1.md | 6 + docs/news/posts/blogger-post-2008-05-23-2.md | 6 + docs/news/posts/blogger-post-2008-05-24-1.md | 6 + docs/news/posts/blogger-post-2008-05-26-1.md | 6 + docs/news/posts/blogger-post-2008-06-09-1.md | 6 + docs/news/posts/blogger-post-2008-09-01-1.md | 6 + docs/news/posts/blogger-post-2008-09-11-1.md | 6 + docs/news/posts/blogger-post-2008-09-19-1.md | 6 + docs/news/posts/blogger-post-2008-09-28-1.md | 6 + docs/news/posts/blogger-post-2008-10-07-1.md | 6 + docs/news/posts/blogger-post-2008-10-14-1.md | 6 + docs/news/posts/blogger-post-2008-10-18-1.md | 6 + docs/news/posts/blogger-post-2008-10-19-1.md | 6 + docs/news/posts/blogger-post-2008-10-23-1.md | 6 + docs/news/posts/blogger-post-2008-11-01-1.md | 6 + docs/news/posts/blogger-post-2009-01-27-1.md | 6 + docs/news/posts/blogger-post-2009-02-04-1.md | 6 + docs/news/posts/blogger-post-2009-02-06-1.md | 6 + docs/news/posts/blogger-post-2009-02-25-1.md | 6 + docs/news/posts/blogger-post-2009-03-11-1.md | 6 + docs/news/posts/blogger-post-2009-03-30-1.md | 6 + docs/news/posts/blogger-post-2009-04-07-1.md | 6 + docs/news/posts/blogger-post-2009-04-25-1.md | 6 + docs/news/posts/blogger-post-2009-04-29-1.md | 6 + docs/news/posts/blogger-post-2009-05-06-1.md | 6 + docs/news/posts/blogger-post-2009-05-07-1.md | 6 + docs/news/posts/blogger-post-2009-06-12-1.md | 6 + docs/news/posts/blogger-post-2009-06-18-1.md | 6 + docs/news/posts/blogger-post-2009-07-24-1.md | 6 + docs/news/posts/blogger-post-2009-08-06-1.md | 6 + docs/news/posts/blogger-post-2009-08-20-1.md | 6 + docs/news/posts/blogger-post-2009-08-22-1.md | 6 + docs/news/posts/blogger-post-2009-08-27-1.md | 6 + docs/news/posts/blogger-post-2009-09-06-1.md | 7 + docs/news/posts/blogger-post-2009-09-16-1.md | 12 + docs/news/posts/blogger-post-2009-11-17-1.md | 6 + docs/news/posts/blogger-post-2009-12-21-1.md | 6 + docs/news/posts/blogger-post-2010-02-13-1.md | 6 + docs/news/posts/blogger-post-2010-04-05-1.md | 6 + docs/news/posts/blogger-post-2010-05-04-1.md | 6 + docs/news/posts/blogger-post-2010-05-26-1.md | 6 + docs/news/posts/blogger-post-2010-05-27-1.md | 6 + docs/news/posts/blogger-post-2010-06-14-1.md | 6 + docs/news/posts/blogger-post-2010-06-21-1.md | 6 + docs/news/posts/blogger-post-2010-06-24-1.md | 6 + docs/news/posts/blogger-post-2010-07-03-1.md | 6 + docs/news/posts/blogger-post-2010-07-13-1.md | 6 + docs/news/posts/blogger-post-2010-07-19-1.md | 6 + docs/news/posts/blogger-post-2010-07-31-1.md | 6 + docs/news/posts/blogger-post-2010-08-13-1.md | 6 + docs/news/posts/blogger-post-2010-08-14-1.md | 6 + docs/news/posts/blogger-post-2010-08-21-1.md | 6 + docs/news/posts/blogger-post-2010-09-02-1.md | 6 + docs/news/posts/blogger-post-2010-10-10-1.md | 6 + docs/news/posts/blogger-post-2010-10-10-2.md | 6 + docs/news/posts/blogger-post-2010-10-20-1.md | 6 + docs/news/posts/blogger-post-2010-10-21-1.md | 6 + docs/news/posts/blogger-post-2010-11-10-1.md | 6 + docs/news/posts/blogger-post-2010-11-17-1.md | 6 + docs/news/posts/blogger-post-2010-11-23-1.md | 6 + docs/news/posts/blogger-post-2010-12-10-1.md | 6 + docs/news/posts/blogger-post-2011-01-17-1.md | 6 + docs/news/posts/blogger-post-2011-01-19-1.md | 6 + docs/news/posts/blogger-post-2011-02-10-1.md | 6 + docs/news/posts/blogger-post-2011-03-02-1.md | 6 + docs/news/posts/blogger-post-2011-03-24-1.md | 6 + docs/news/posts/blogger-post-2011-03-30-1.md | 6 + docs/news/posts/blogger-post-2011-04-05-1.md | 6 + docs/news/posts/blogger-post-2011-04-15-1.md | 6 + docs/news/posts/blogger-post-2011-04-17-1.md | 6 + docs/news/posts/blogger-post-2011-04-24-1.md | 6 + docs/news/posts/blogger-post-2011-05-03-1.md | 6 + docs/news/posts/blogger-post-2011-05-12-1.md | 6 + docs/news/posts/blogger-post-2011-06-01-1.md | 6 + docs/news/posts/blogger-post-2011-06-05-1.md | 6 + docs/news/posts/blogger-post-2011-06-13-1.md | 10 + docs/news/posts/blogger-post-2011-06-25-1.md | 6 + docs/news/posts/blogger-post-2011-08-15-1.md | 6 + docs/news/posts/blogger-post-2011-08-18-1.md | 8 + docs/news/posts/blogger-post-2011-08-19-1.md | 6 + docs/news/posts/blogger-post-2011-09-11-1.md | 6 + docs/news/posts/blogger-post-2011-09-18-1.md | 6 + docs/news/posts/blogger-post-2011-09-19-1.md | 6 + docs/news/posts/blogger-post-2011-09-21-1.md | 6 + docs/news/posts/blogger-post-2011-10-10-1.md | 6 + docs/news/posts/blogger-post-2011-11-01-1.md | 6 + docs/news/posts/blogger-post-2011-11-18-1.md | 6 + docs/news/posts/blogger-post-2011-12-07-1.md | 6 + docs/news/posts/blogger-post-2011-12-17-1.md | 6 + docs/news/posts/blogger-post-2012-01-26-1.md | 6 + docs/news/posts/blogger-post-2012-02-18-1.md | 13 + docs/news/posts/blogger-post-2012-03-11-1.md | 8 + docs/news/posts/blogger-post-2012-05-23-1.md | 8 + docs/news/posts/blogger-post-2012-06-11-1.md | 8 + docs/news/posts/blogger-post-2012-06-12-1.md | 70 ++++ docs/news/posts/blogger-post-2012-07-02-1.md | 11 + docs/news/posts/blogger-post-2012-07-06-1.md | 15 + docs/news/posts/blogger-post-2012-07-22-1.md | 9 + docs/news/posts/blogger-post-2012-07-24-1.md | 7 + docs/news/posts/blogger-post-2012-07-29-1.md | 19 + docs/news/posts/blogger-post-2012-08-04-1.md | 9 + docs/news/posts/blogger-post-2012-08-12-1.md | 11 + docs/news/posts/blogger-post-2012-08-15-1.md | 8 + docs/news/posts/blogger-post-2012-08-16-1.md | 12 + docs/news/posts/blogger-post-2012-08-30-1.md | 8 + docs/news/posts/blogger-post-2012-09-04-1.md | 418 +++++++++++++++++++ docs/news/posts/blogger-post-2012-09-07-1.md | 12 + docs/news/posts/blogger-post-2012-10-07-1.md | 29 ++ docs/news/posts/blogger-post-2012-10-28-1.md | 10 + docs/news/posts/blogger-post-2012-11-10-1.md | 12 + docs/news/posts/blogger-post-2012-11-28-1.md | 12 + docs/news/posts/blogger-post-2012-11-29-1.md | 12 + docs/news/posts/blogger-post-2012-12-03-1.md | 8 + docs/news/posts/blogger-post-2013-01-18-1.md | 6 + docs/news/posts/blogger-post-2013-02-17-1.md | 18 + docs/news/posts/blogger-post-2013-02-17-2.md | 18 + docs/news/posts/blogger-post-2013-04-12-1.md | 6 + docs/news/posts/blogger-post-2013-04-28-1.md | 6 + docs/news/posts/blogger-post-2013-06-30-1.md | 10 + docs/news/posts/blogger-post-2013-07-15-1.md | 12 + docs/news/posts/blogger-post-2013-08-12-1.md | 20 + docs/news/posts/blogger-post-2013-09-03-1.md | 13 + docs/news/posts/blogger-post-2013-09-03-2.md | 12 + docs/news/posts/blogger-post-2013-10-01-1.md | 9 + docs/news/posts/blogger-post-2013-10-07-1.md | 8 + docs/news/posts/blogger-post-2013-10-07-2.md | 27 ++ docs/news/posts/blogger-post-2013-10-11-1.md | 10 + docs/news/posts/blogger-post-2013-11-10-1.md | 16 + docs/news/posts/blogger-post-2013-11-10-2.md | 18 + docs/news/posts/blogger-post-2013-11-21-1.md | 48 +++ docs/news/posts/blogger-post-2014-03-08-1.md | 8 + docs/news/posts/blogger-post-2014-03-15-1.md | 6 + docs/news/posts/blogger-post-2014-03-18-1.md | 8 + docs/news/posts/blogger-post-2014-03-19-1.md | 10 + docs/news/posts/blogger-post-2014-04-08-1.md | 6 + docs/news/posts/blogger-post-2014-04-10-1.md | 6 + docs/news/posts/blogger-post-2014-04-10-2.md | 18 + docs/news/posts/blogger-post-2014-04-27-1.md | 6 + docs/news/posts/blogger-post-2014-06-01-1.md | 20 + docs/news/posts/blogger-post-2014-06-04-1.md | 17 + docs/news/posts/blogger-post-2014-06-07-1.md | 6 + docs/news/posts/blogger-post-2014-06-17-1.md | 18 + docs/news/posts/blogger-post-2014-07-15-1.md | 6 + docs/news/posts/blogger-post-2014-08-01-1.md | 10 + docs/news/posts/blogger-post-2014-08-05-1.md | 8 + docs/news/posts/blogger-post-2014-08-11-1.md | 10 + docs/news/posts/blogger-post-2014-08-11-2.md | 10 + docs/news/posts/blogger-post-2014-10-10-1.md | 10 + docs/news/posts/blogger-post-2014-10-24-1.md | 28 ++ docs/news/posts/blogger-post-2014-10-26-1.md | 12 + docs/news/posts/blogger-post-2014-11-04-1.md | 8 + docs/news/posts/blogger-post-2014-12-01-1.md | 13 + docs/news/posts/blogger-post-2014-12-10-1.md | 29 ++ docs/news/posts/blogger-post-2014-12-26-1.md | 31 ++ docs/news/posts/blogger-post-2015-01-14-1.md | 8 + docs/news/posts/blogger-post-2015-01-27-1.md | 6 + docs/news/posts/blogger-post-2015-01-29-1.md | 6 + docs/news/posts/blogger-post-2015-02-12-1.md | 6 + docs/news/posts/blogger-post-2015-05-05-1.md | 12 + docs/news/posts/blogger-post-2015-05-09-1.md | 16 + docs/news/posts/blogger-post-2015-05-20-1.md | 6 + docs/news/posts/blogger-post-2015-05-20-2.md | 6 + docs/news/posts/blogger-post-2015-05-20-3.md | 6 + docs/news/posts/blogger-post-2015-05-21-1.md | 6 + docs/news/posts/blogger-post-2015-05-21-2.md | 14 + docs/news/posts/blogger-post-2015-06-15-1.md | 8 + docs/news/posts/blogger-post-2015-06-15-2.md | 6 + docs/news/posts/blogger-post-2015-06-20-1.md | 34 ++ docs/news/posts/blogger-post-2015-08-09-1.md | 10 + docs/news/posts/blogger-post-2015-08-30-1.md | 14 + docs/news/posts/blogger-post-2015-09-19-1.md | 17 + docs/news/posts/blogger-post-2015-09-20-1.md | 12 + docs/news/posts/blogger-post-2015-10-09-1.md | 6 + docs/news/posts/blogger-post-2015-10-12-1.md | 20 + docs/news/posts/blogger-post-2015-10-18-1.md | 6 + docs/news/posts/blogger-post-2015-11-06-1.md | 6 + docs/news/posts/blogger-post-2015-11-09-1.md | 10 + docs/news/posts/blogger-post-2015-11-09-2.md | 8 + docs/news/posts/blogger-post-2015-11-14-1.md | 6 + docs/news/posts/blogger-post-2015-12-04-1.md | 6 + docs/news/posts/blogger-post-2015-12-09-1.md | 6 + docs/news/posts/blogger-post-2015-12-17-1.md | 7 + docs/news/posts/blogger-post-2016-01-19-1.md | 8 + docs/news/posts/blogger-post-2016-01-21-1.md | 8 + docs/news/posts/blogger-post-2016-01-22-1.md | 8 + docs/news/posts/blogger-post-2016-02-01-1.md | 13 + docs/news/posts/blogger-post-2016-02-05-1.md | 9 + docs/news/posts/blogger-post-2016-02-06-1.md | 33 ++ docs/news/posts/blogger-post-2016-02-12-1.md | 8 + docs/news/posts/blogger-post-2016-02-13-1.md | 14 + docs/news/posts/blogger-post-2016-02-16-1.md | 6 + docs/news/posts/blogger-post-2016-02-29-1.md | 12 + docs/news/posts/blogger-post-2016-03-11-1.md | 6 + docs/news/posts/blogger-post-2016-03-11-2.md | 29 ++ docs/news/posts/blogger-post-2016-03-19-1.md | 37 ++ docs/news/posts/blogger-post-2016-03-29-1.md | 6 + docs/news/posts/blogger-post-2016-03-30-1.md | 36 ++ docs/news/posts/blogger-post-2016-04-15-1.md | 8 + docs/news/posts/blogger-post-2016-05-15-1.md | 10 + docs/news/posts/blogger-post-2016-05-26-1.md | 8 + docs/news/posts/blogger-post-2016-07-18-1.md | 28 ++ docs/news/posts/blogger-post-2016-08-24-1.md | 22 + docs/news/posts/blogger-post-2016-08-26-1.md | 8 + docs/news/posts/blogger-post-2016-09-16-1.md | 14 + docs/news/posts/blogger-post-2016-09-16-2.md | 10 + docs/news/posts/blogger-post-2016-09-20-1.md | 16 + docs/news/posts/blogger-post-2016-09-22-1.md | 10 + docs/news/posts/blogger-post-2016-10-01-1.md | 37 ++ docs/news/posts/blogger-post-2016-10-20-1.md | 6 + docs/news/posts/blogger-post-2016-11-07-1.md | 8 + docs/news/posts/blogger-post-2016-11-08-1.md | 7 + docs/news/posts/blogger-post-2016-11-21-1.md | 12 + docs/news/posts/blogger-post-2016-12-03-1.md | 8 + docs/news/posts/blogger-post-2017-02-18-1.md | 23 + docs/news/posts/blogger-post-2017-02-19-1.md | 14 + docs/news/posts/blogger-post-2017-03-17-1.md | 10 + docs/news/posts/blogger-post-2017-04-20-1.md | 8 + docs/news/posts/blogger-post-2017-05-31-1.md | 28 ++ docs/news/posts/blogger-post-2017-07-08-1.md | 6 + docs/news/posts/blogger-post-2017-08-02-1.md | 11 + docs/news/posts/blogger-post-2017-10-15-1.md | 6 + docs/news/posts/blogger-post-2017-12-24-1.md | 6 + docs/news/posts/blogger-post-2018-01-17-1.md | 20 + docs/news/posts/blogger-post-2018-02-10-1.md | 8 + docs/news/posts/blogger-post-2018-03-20-1.md | 11 + docs/news/posts/blogger-post-2018-06-02-1.md | 15 + docs/news/posts/blogger-post-2018-10-04-1.md | 15 + docs/news/posts/blogger-post-2018-10-08-1.md | 16 + docs/news/posts/blogger-post-2018-10-12-1.md | 8 + docs/news/posts/blogger-post-2018-10-12-2.md | 6 + docs/news/posts/blogger-post-2018-11-07-1.md | 37 ++ docs/news/posts/blogger-post-2018-12-21-1.md | 8 + docs/news/posts/blogger-post-2019-01-04-1.md | 17 + docs/news/posts/blogger-post-2019-05-03-1.md | 16 + docs/news/posts/blogger-post-2019-05-23-1.md | 14 + docs/news/posts/blogger-post-2019-06-14-1.md | 10 + docs/news/posts/blogger-post-2019-09-20-1.md | 9 + docs/news/posts/blogger-post-2019-10-01-1.md | 8 + docs/news/posts/blogger-post-2019-10-09-1.md | 17 + docs/news/posts/blogger-post-2019-10-23-1.md | 12 + docs/news/posts/blogger-post-2019-10-28-1.md | 12 + docs/news/posts/blogger-post-2019-11-07-1.md | 8 + docs/news/posts/blogger-post-2019-11-12-1.md | 6 + docs/news/posts/blogger-post-2020-01-24-1.md | 13 + docs/news/posts/blogger-post-2020-03-07-1.md | 6 + docs/news/posts/blogger-post-2020-03-10-1.md | 13 + docs/news/posts/blogger-post-2020-03-18-1.md | 13 + docs/news/posts/blogger-post-2020-04-03-1.md | 12 + docs/news/posts/blogger-post-2020-05-18-1.md | 10 + docs/news/posts/blogger-post-2020-07-04-1.md | 8 + docs/news/posts/blogger-post-2020-07-26-1.md | 10 + docs/news/posts/blogger-post-2020-09-17-1.md | 8 + docs/news/posts/blogger-post-2020-09-28-1.md | 6 + docs/news/posts/blogger-post-2020-10-03-1.md | 6 + docs/news/posts/blogger-post-2021-05-31-1.md | 6 + docs/news/posts/blogger-post-2021-06-24-1.md | 6 + docs/news/posts/blogger-post-2021-07-05-1.md | 6 + docs/news/posts/blogger-post-2021-09-23-1.md | 6 + docs/news/posts/blogger-post-2021-10-08-1.md | 6 + docs/news/posts/blogger-post-2021-10-08-2.md | 6 + docs/news/posts/blogger-post-2021-10-11-1.md | 6 + docs/news/posts/blogger-post-2021-10-23-1.md | 6 + docs/news/posts/blogger-post-2021-10-24-1.md | 6 + docs/news/posts/blogger-post-2021-11-02-1.md | 6 + docs/news/posts/blogger-post-2021-11-14-1.md | 6 + docs/news/posts/blogger-post-2021-11-18-1.md | 6 + docs/news/posts/blogger-post-2021-12-20-1.md | 6 + docs/news/posts/blogger-post-2021-12-20-2.md | 6 + docs/news/posts/blogger-post-2022-01-21-1.md | 6 + docs/news/posts/blogger-post-2022-02-05-1.md | 6 + docs/news/posts/blogger-post-2022-02-07-1.md | 6 + docs/news/posts/blogger-post-2022-02-08-1.md | 6 + docs/news/posts/blogger-post-2022-02-14-1.md | 6 + docs/news/posts/blogger-post-2022-03-11-1.md | 6 + docs/news/posts/blogger-post-2022-03-13-1.md | 6 + docs/news/posts/blogger-post-2022-03-15-1.md | 6 + docs/news/posts/blogger-post-2022-04-03-1.md | 6 + docs/news/posts/blogger-post-2022-05-13-1.md | 6 + docs/news/posts/blogger-post-2022-08-27-1.md | 6 + docs/news/posts/blogger-post-2022-09-11-1.md | 6 + docs/news/posts/blogger-post-2023-01-18-1.md | 6 + docs/news/posts/blogger-post-2024-02-09-1.md | 6 + docs/news/posts/blogger-post-2024-08-11-1.md | 6 + docs/news/posts/blogger-post-2024-08-11-2.md | 6 + 355 files changed, 3658 insertions(+) create mode 100644 docs/news/posts/blogger-post-2006-09-23-1.md create mode 100644 docs/news/posts/blogger-post-2006-09-25-1.md create mode 100644 docs/news/posts/blogger-post-2006-09-26-1.md create mode 100644 docs/news/posts/blogger-post-2006-09-29-1.md create mode 100644 docs/news/posts/blogger-post-2006-10-02-1.md create mode 100644 docs/news/posts/blogger-post-2006-10-06-1.md create mode 100644 docs/news/posts/blogger-post-2006-10-12-1.md create mode 100644 docs/news/posts/blogger-post-2006-10-12-2.md create mode 100644 docs/news/posts/blogger-post-2006-10-12-3.md create mode 100644 docs/news/posts/blogger-post-2006-10-13-1.md create mode 100644 docs/news/posts/blogger-post-2006-10-13-2.md create mode 100644 docs/news/posts/blogger-post-2006-10-14-1.md create mode 100644 docs/news/posts/blogger-post-2006-10-15-1.md create mode 100644 docs/news/posts/blogger-post-2006-10-16-1.md create mode 100644 docs/news/posts/blogger-post-2006-10-16-2.md create mode 100644 docs/news/posts/blogger-post-2006-10-16-3.md create mode 100644 docs/news/posts/blogger-post-2006-10-17-1.md create mode 100644 docs/news/posts/blogger-post-2006-10-17-2.md create mode 100644 docs/news/posts/blogger-post-2006-10-18-1.md create mode 100644 docs/news/posts/blogger-post-2006-10-19-1.md create mode 100644 docs/news/posts/blogger-post-2006-10-22-1.md create mode 100644 docs/news/posts/blogger-post-2006-10-26-1.md create mode 100644 docs/news/posts/blogger-post-2006-10-28-1.md create mode 100644 docs/news/posts/blogger-post-2006-10-29-1.md create mode 100644 docs/news/posts/blogger-post-2006-11-05-1.md create mode 100644 docs/news/posts/blogger-post-2006-11-07-1.md create mode 100644 docs/news/posts/blogger-post-2006-11-11-1.md create mode 100644 docs/news/posts/blogger-post-2006-11-11-2.md create mode 100644 docs/news/posts/blogger-post-2006-11-15-1.md create mode 100644 docs/news/posts/blogger-post-2006-11-16-1.md create mode 100644 docs/news/posts/blogger-post-2006-11-20-1.md create mode 100644 docs/news/posts/blogger-post-2006-11-23-1.md create mode 100644 docs/news/posts/blogger-post-2006-11-29-1.md create mode 100644 docs/news/posts/blogger-post-2006-11-30-1.md create mode 100644 docs/news/posts/blogger-post-2006-12-10-1.md create mode 100644 docs/news/posts/blogger-post-2007-02-04-1.md create mode 100644 docs/news/posts/blogger-post-2007-02-04-2.md create mode 100644 docs/news/posts/blogger-post-2007-02-05-1.md create mode 100644 docs/news/posts/blogger-post-2007-02-26-1.md create mode 100644 docs/news/posts/blogger-post-2007-03-04-1.md create mode 100644 docs/news/posts/blogger-post-2007-03-06-1.md create mode 100644 docs/news/posts/blogger-post-2007-03-11-1.md create mode 100644 docs/news/posts/blogger-post-2007-03-24-1.md create mode 100644 docs/news/posts/blogger-post-2007-03-25-1.md create mode 100644 docs/news/posts/blogger-post-2007-03-29-1.md create mode 100644 docs/news/posts/blogger-post-2007-05-02-1.md create mode 100644 docs/news/posts/blogger-post-2007-05-20-1.md create mode 100644 docs/news/posts/blogger-post-2007-06-01-1.md create mode 100644 docs/news/posts/blogger-post-2007-06-15-1.md create mode 100644 docs/news/posts/blogger-post-2007-06-19-1.md create mode 100644 docs/news/posts/blogger-post-2007-06-30-1.md create mode 100644 docs/news/posts/blogger-post-2007-07-02-1.md create mode 100644 docs/news/posts/blogger-post-2007-08-01-1.md create mode 100644 docs/news/posts/blogger-post-2007-08-04-1.md create mode 100644 docs/news/posts/blogger-post-2007-09-07-1.md create mode 100644 docs/news/posts/blogger-post-2007-09-22-1.md create mode 100644 docs/news/posts/blogger-post-2007-10-08-1.md create mode 100644 docs/news/posts/blogger-post-2007-10-08-2.md create mode 100644 docs/news/posts/blogger-post-2007-10-18-1.md create mode 100644 docs/news/posts/blogger-post-2007-10-18-2.md create mode 100644 docs/news/posts/blogger-post-2007-11-03-1.md create mode 100644 docs/news/posts/blogger-post-2008-01-31-1.md create mode 100644 docs/news/posts/blogger-post-2008-02-29-1.md create mode 100644 docs/news/posts/blogger-post-2008-03-12-1.md create mode 100644 docs/news/posts/blogger-post-2008-03-15-1.md create mode 100644 docs/news/posts/blogger-post-2008-03-18-1.md create mode 100644 docs/news/posts/blogger-post-2008-03-21-1.md create mode 100644 docs/news/posts/blogger-post-2008-03-25-1.md create mode 100644 docs/news/posts/blogger-post-2008-04-18-1.md create mode 100644 docs/news/posts/blogger-post-2008-05-17-1.md create mode 100644 docs/news/posts/blogger-post-2008-05-20-1.md create mode 100644 docs/news/posts/blogger-post-2008-05-23-1.md create mode 100644 docs/news/posts/blogger-post-2008-05-23-2.md create mode 100644 docs/news/posts/blogger-post-2008-05-24-1.md create mode 100644 docs/news/posts/blogger-post-2008-05-26-1.md create mode 100644 docs/news/posts/blogger-post-2008-06-09-1.md create mode 100644 docs/news/posts/blogger-post-2008-09-01-1.md create mode 100644 docs/news/posts/blogger-post-2008-09-11-1.md create mode 100644 docs/news/posts/blogger-post-2008-09-19-1.md create mode 100644 docs/news/posts/blogger-post-2008-09-28-1.md create mode 100644 docs/news/posts/blogger-post-2008-10-07-1.md create mode 100644 docs/news/posts/blogger-post-2008-10-14-1.md create mode 100644 docs/news/posts/blogger-post-2008-10-18-1.md create mode 100644 docs/news/posts/blogger-post-2008-10-19-1.md create mode 100644 docs/news/posts/blogger-post-2008-10-23-1.md create mode 100644 docs/news/posts/blogger-post-2008-11-01-1.md create mode 100644 docs/news/posts/blogger-post-2009-01-27-1.md create mode 100644 docs/news/posts/blogger-post-2009-02-04-1.md create mode 100644 docs/news/posts/blogger-post-2009-02-06-1.md create mode 100644 docs/news/posts/blogger-post-2009-02-25-1.md create mode 100644 docs/news/posts/blogger-post-2009-03-11-1.md create mode 100644 docs/news/posts/blogger-post-2009-03-30-1.md create mode 100644 docs/news/posts/blogger-post-2009-04-07-1.md create mode 100644 docs/news/posts/blogger-post-2009-04-25-1.md create mode 100644 docs/news/posts/blogger-post-2009-04-29-1.md create mode 100644 docs/news/posts/blogger-post-2009-05-06-1.md create mode 100644 docs/news/posts/blogger-post-2009-05-07-1.md create mode 100644 docs/news/posts/blogger-post-2009-06-12-1.md create mode 100644 docs/news/posts/blogger-post-2009-06-18-1.md create mode 100644 docs/news/posts/blogger-post-2009-07-24-1.md create mode 100644 docs/news/posts/blogger-post-2009-08-06-1.md create mode 100644 docs/news/posts/blogger-post-2009-08-20-1.md create mode 100644 docs/news/posts/blogger-post-2009-08-22-1.md create mode 100644 docs/news/posts/blogger-post-2009-08-27-1.md create mode 100644 docs/news/posts/blogger-post-2009-09-06-1.md create mode 100644 docs/news/posts/blogger-post-2009-09-16-1.md create mode 100644 docs/news/posts/blogger-post-2009-11-17-1.md create mode 100644 docs/news/posts/blogger-post-2009-12-21-1.md create mode 100644 docs/news/posts/blogger-post-2010-02-13-1.md create mode 100644 docs/news/posts/blogger-post-2010-04-05-1.md create mode 100644 docs/news/posts/blogger-post-2010-05-04-1.md create mode 100644 docs/news/posts/blogger-post-2010-05-26-1.md create mode 100644 docs/news/posts/blogger-post-2010-05-27-1.md create mode 100644 docs/news/posts/blogger-post-2010-06-14-1.md create mode 100644 docs/news/posts/blogger-post-2010-06-21-1.md create mode 100644 docs/news/posts/blogger-post-2010-06-24-1.md create mode 100644 docs/news/posts/blogger-post-2010-07-03-1.md create mode 100644 docs/news/posts/blogger-post-2010-07-13-1.md create mode 100644 docs/news/posts/blogger-post-2010-07-19-1.md create mode 100644 docs/news/posts/blogger-post-2010-07-31-1.md create mode 100644 docs/news/posts/blogger-post-2010-08-13-1.md create mode 100644 docs/news/posts/blogger-post-2010-08-14-1.md create mode 100644 docs/news/posts/blogger-post-2010-08-21-1.md create mode 100644 docs/news/posts/blogger-post-2010-09-02-1.md create mode 100644 docs/news/posts/blogger-post-2010-10-10-1.md create mode 100644 docs/news/posts/blogger-post-2010-10-10-2.md create mode 100644 docs/news/posts/blogger-post-2010-10-20-1.md create mode 100644 docs/news/posts/blogger-post-2010-10-21-1.md create mode 100644 docs/news/posts/blogger-post-2010-11-10-1.md create mode 100644 docs/news/posts/blogger-post-2010-11-17-1.md create mode 100644 docs/news/posts/blogger-post-2010-11-23-1.md create mode 100644 docs/news/posts/blogger-post-2010-12-10-1.md create mode 100644 docs/news/posts/blogger-post-2011-01-17-1.md create mode 100644 docs/news/posts/blogger-post-2011-01-19-1.md create mode 100644 docs/news/posts/blogger-post-2011-02-10-1.md create mode 100644 docs/news/posts/blogger-post-2011-03-02-1.md create mode 100644 docs/news/posts/blogger-post-2011-03-24-1.md create mode 100644 docs/news/posts/blogger-post-2011-03-30-1.md create mode 100644 docs/news/posts/blogger-post-2011-04-05-1.md create mode 100644 docs/news/posts/blogger-post-2011-04-15-1.md create mode 100644 docs/news/posts/blogger-post-2011-04-17-1.md create mode 100644 docs/news/posts/blogger-post-2011-04-24-1.md create mode 100644 docs/news/posts/blogger-post-2011-05-03-1.md create mode 100644 docs/news/posts/blogger-post-2011-05-12-1.md create mode 100644 docs/news/posts/blogger-post-2011-06-01-1.md create mode 100644 docs/news/posts/blogger-post-2011-06-05-1.md create mode 100644 docs/news/posts/blogger-post-2011-06-13-1.md create mode 100644 docs/news/posts/blogger-post-2011-06-25-1.md create mode 100644 docs/news/posts/blogger-post-2011-08-15-1.md create mode 100644 docs/news/posts/blogger-post-2011-08-18-1.md create mode 100644 docs/news/posts/blogger-post-2011-08-19-1.md create mode 100644 docs/news/posts/blogger-post-2011-09-11-1.md create mode 100644 docs/news/posts/blogger-post-2011-09-18-1.md create mode 100644 docs/news/posts/blogger-post-2011-09-19-1.md create mode 100644 docs/news/posts/blogger-post-2011-09-21-1.md create mode 100644 docs/news/posts/blogger-post-2011-10-10-1.md create mode 100644 docs/news/posts/blogger-post-2011-11-01-1.md create mode 100644 docs/news/posts/blogger-post-2011-11-18-1.md create mode 100644 docs/news/posts/blogger-post-2011-12-07-1.md create mode 100644 docs/news/posts/blogger-post-2011-12-17-1.md create mode 100644 docs/news/posts/blogger-post-2012-01-26-1.md create mode 100644 docs/news/posts/blogger-post-2012-02-18-1.md create mode 100644 docs/news/posts/blogger-post-2012-03-11-1.md create mode 100644 docs/news/posts/blogger-post-2012-05-23-1.md create mode 100644 docs/news/posts/blogger-post-2012-06-11-1.md create mode 100644 docs/news/posts/blogger-post-2012-06-12-1.md create mode 100644 docs/news/posts/blogger-post-2012-07-02-1.md create mode 100644 docs/news/posts/blogger-post-2012-07-06-1.md create mode 100644 docs/news/posts/blogger-post-2012-07-22-1.md create mode 100644 docs/news/posts/blogger-post-2012-07-24-1.md create mode 100644 docs/news/posts/blogger-post-2012-07-29-1.md create mode 100644 docs/news/posts/blogger-post-2012-08-04-1.md create mode 100644 docs/news/posts/blogger-post-2012-08-12-1.md create mode 100644 docs/news/posts/blogger-post-2012-08-15-1.md create mode 100644 docs/news/posts/blogger-post-2012-08-16-1.md create mode 100644 docs/news/posts/blogger-post-2012-08-30-1.md create mode 100644 docs/news/posts/blogger-post-2012-09-04-1.md create mode 100644 docs/news/posts/blogger-post-2012-09-07-1.md create mode 100644 docs/news/posts/blogger-post-2012-10-07-1.md create mode 100644 docs/news/posts/blogger-post-2012-10-28-1.md create mode 100644 docs/news/posts/blogger-post-2012-11-10-1.md create mode 100644 docs/news/posts/blogger-post-2012-11-28-1.md create mode 100644 docs/news/posts/blogger-post-2012-11-29-1.md create mode 100644 docs/news/posts/blogger-post-2012-12-03-1.md create mode 100644 docs/news/posts/blogger-post-2013-01-18-1.md create mode 100644 docs/news/posts/blogger-post-2013-02-17-1.md create mode 100644 docs/news/posts/blogger-post-2013-02-17-2.md create mode 100644 docs/news/posts/blogger-post-2013-04-12-1.md create mode 100644 docs/news/posts/blogger-post-2013-04-28-1.md create mode 100644 docs/news/posts/blogger-post-2013-06-30-1.md create mode 100644 docs/news/posts/blogger-post-2013-07-15-1.md create mode 100644 docs/news/posts/blogger-post-2013-08-12-1.md create mode 100644 docs/news/posts/blogger-post-2013-09-03-1.md create mode 100644 docs/news/posts/blogger-post-2013-09-03-2.md create mode 100644 docs/news/posts/blogger-post-2013-10-01-1.md create mode 100644 docs/news/posts/blogger-post-2013-10-07-1.md create mode 100644 docs/news/posts/blogger-post-2013-10-07-2.md create mode 100644 docs/news/posts/blogger-post-2013-10-11-1.md create mode 100644 docs/news/posts/blogger-post-2013-11-10-1.md create mode 100644 docs/news/posts/blogger-post-2013-11-10-2.md create mode 100644 docs/news/posts/blogger-post-2013-11-21-1.md create mode 100644 docs/news/posts/blogger-post-2014-03-08-1.md create mode 100644 docs/news/posts/blogger-post-2014-03-15-1.md create mode 100644 docs/news/posts/blogger-post-2014-03-18-1.md create mode 100644 docs/news/posts/blogger-post-2014-03-19-1.md create mode 100644 docs/news/posts/blogger-post-2014-04-08-1.md create mode 100644 docs/news/posts/blogger-post-2014-04-10-1.md create mode 100644 docs/news/posts/blogger-post-2014-04-10-2.md create mode 100644 docs/news/posts/blogger-post-2014-04-27-1.md create mode 100644 docs/news/posts/blogger-post-2014-06-01-1.md create mode 100644 docs/news/posts/blogger-post-2014-06-04-1.md create mode 100644 docs/news/posts/blogger-post-2014-06-07-1.md create mode 100644 docs/news/posts/blogger-post-2014-06-17-1.md create mode 100644 docs/news/posts/blogger-post-2014-07-15-1.md create mode 100644 docs/news/posts/blogger-post-2014-08-01-1.md create mode 100644 docs/news/posts/blogger-post-2014-08-05-1.md create mode 100644 docs/news/posts/blogger-post-2014-08-11-1.md create mode 100644 docs/news/posts/blogger-post-2014-08-11-2.md create mode 100644 docs/news/posts/blogger-post-2014-10-10-1.md create mode 100644 docs/news/posts/blogger-post-2014-10-24-1.md create mode 100644 docs/news/posts/blogger-post-2014-10-26-1.md create mode 100644 docs/news/posts/blogger-post-2014-11-04-1.md create mode 100644 docs/news/posts/blogger-post-2014-12-01-1.md create mode 100644 docs/news/posts/blogger-post-2014-12-10-1.md create mode 100644 docs/news/posts/blogger-post-2014-12-26-1.md create mode 100644 docs/news/posts/blogger-post-2015-01-14-1.md create mode 100644 docs/news/posts/blogger-post-2015-01-27-1.md create mode 100644 docs/news/posts/blogger-post-2015-01-29-1.md create mode 100644 docs/news/posts/blogger-post-2015-02-12-1.md create mode 100644 docs/news/posts/blogger-post-2015-05-05-1.md create mode 100644 docs/news/posts/blogger-post-2015-05-09-1.md create mode 100644 docs/news/posts/blogger-post-2015-05-20-1.md create mode 100644 docs/news/posts/blogger-post-2015-05-20-2.md create mode 100644 docs/news/posts/blogger-post-2015-05-20-3.md create mode 100644 docs/news/posts/blogger-post-2015-05-21-1.md create mode 100644 docs/news/posts/blogger-post-2015-05-21-2.md create mode 100644 docs/news/posts/blogger-post-2015-06-15-1.md create mode 100644 docs/news/posts/blogger-post-2015-06-15-2.md create mode 100644 docs/news/posts/blogger-post-2015-06-20-1.md create mode 100644 docs/news/posts/blogger-post-2015-08-09-1.md create mode 100644 docs/news/posts/blogger-post-2015-08-30-1.md create mode 100644 docs/news/posts/blogger-post-2015-09-19-1.md create mode 100644 docs/news/posts/blogger-post-2015-09-20-1.md create mode 100644 docs/news/posts/blogger-post-2015-10-09-1.md create mode 100644 docs/news/posts/blogger-post-2015-10-12-1.md create mode 100644 docs/news/posts/blogger-post-2015-10-18-1.md create mode 100644 docs/news/posts/blogger-post-2015-11-06-1.md create mode 100644 docs/news/posts/blogger-post-2015-11-09-1.md create mode 100644 docs/news/posts/blogger-post-2015-11-09-2.md create mode 100644 docs/news/posts/blogger-post-2015-11-14-1.md create mode 100644 docs/news/posts/blogger-post-2015-12-04-1.md create mode 100644 docs/news/posts/blogger-post-2015-12-09-1.md create mode 100644 docs/news/posts/blogger-post-2015-12-17-1.md create mode 100644 docs/news/posts/blogger-post-2016-01-19-1.md create mode 100644 docs/news/posts/blogger-post-2016-01-21-1.md create mode 100644 docs/news/posts/blogger-post-2016-01-22-1.md create mode 100644 docs/news/posts/blogger-post-2016-02-01-1.md create mode 100644 docs/news/posts/blogger-post-2016-02-05-1.md create mode 100644 docs/news/posts/blogger-post-2016-02-06-1.md create mode 100644 docs/news/posts/blogger-post-2016-02-12-1.md create mode 100644 docs/news/posts/blogger-post-2016-02-13-1.md create mode 100644 docs/news/posts/blogger-post-2016-02-16-1.md create mode 100644 docs/news/posts/blogger-post-2016-02-29-1.md create mode 100644 docs/news/posts/blogger-post-2016-03-11-1.md create mode 100644 docs/news/posts/blogger-post-2016-03-11-2.md create mode 100644 docs/news/posts/blogger-post-2016-03-19-1.md create mode 100644 docs/news/posts/blogger-post-2016-03-29-1.md create mode 100644 docs/news/posts/blogger-post-2016-03-30-1.md create mode 100644 docs/news/posts/blogger-post-2016-04-15-1.md create mode 100644 docs/news/posts/blogger-post-2016-05-15-1.md create mode 100644 docs/news/posts/blogger-post-2016-05-26-1.md create mode 100644 docs/news/posts/blogger-post-2016-07-18-1.md create mode 100644 docs/news/posts/blogger-post-2016-08-24-1.md create mode 100644 docs/news/posts/blogger-post-2016-08-26-1.md create mode 100644 docs/news/posts/blogger-post-2016-09-16-1.md create mode 100644 docs/news/posts/blogger-post-2016-09-16-2.md create mode 100644 docs/news/posts/blogger-post-2016-09-20-1.md create mode 100644 docs/news/posts/blogger-post-2016-09-22-1.md create mode 100644 docs/news/posts/blogger-post-2016-10-01-1.md create mode 100644 docs/news/posts/blogger-post-2016-10-20-1.md create mode 100644 docs/news/posts/blogger-post-2016-11-07-1.md create mode 100644 docs/news/posts/blogger-post-2016-11-08-1.md create mode 100644 docs/news/posts/blogger-post-2016-11-21-1.md create mode 100644 docs/news/posts/blogger-post-2016-12-03-1.md create mode 100644 docs/news/posts/blogger-post-2017-02-18-1.md create mode 100644 docs/news/posts/blogger-post-2017-02-19-1.md create mode 100644 docs/news/posts/blogger-post-2017-03-17-1.md create mode 100644 docs/news/posts/blogger-post-2017-04-20-1.md create mode 100644 docs/news/posts/blogger-post-2017-05-31-1.md create mode 100644 docs/news/posts/blogger-post-2017-07-08-1.md create mode 100644 docs/news/posts/blogger-post-2017-08-02-1.md create mode 100644 docs/news/posts/blogger-post-2017-10-15-1.md create mode 100644 docs/news/posts/blogger-post-2017-12-24-1.md create mode 100644 docs/news/posts/blogger-post-2018-01-17-1.md create mode 100644 docs/news/posts/blogger-post-2018-02-10-1.md create mode 100644 docs/news/posts/blogger-post-2018-03-20-1.md create mode 100644 docs/news/posts/blogger-post-2018-06-02-1.md create mode 100644 docs/news/posts/blogger-post-2018-10-04-1.md create mode 100644 docs/news/posts/blogger-post-2018-10-08-1.md create mode 100644 docs/news/posts/blogger-post-2018-10-12-1.md create mode 100644 docs/news/posts/blogger-post-2018-10-12-2.md create mode 100644 docs/news/posts/blogger-post-2018-11-07-1.md create mode 100644 docs/news/posts/blogger-post-2018-12-21-1.md create mode 100644 docs/news/posts/blogger-post-2019-01-04-1.md create mode 100644 docs/news/posts/blogger-post-2019-05-03-1.md create mode 100644 docs/news/posts/blogger-post-2019-05-23-1.md create mode 100644 docs/news/posts/blogger-post-2019-06-14-1.md create mode 100644 docs/news/posts/blogger-post-2019-09-20-1.md create mode 100644 docs/news/posts/blogger-post-2019-10-01-1.md create mode 100644 docs/news/posts/blogger-post-2019-10-09-1.md create mode 100644 docs/news/posts/blogger-post-2019-10-23-1.md create mode 100644 docs/news/posts/blogger-post-2019-10-28-1.md create mode 100644 docs/news/posts/blogger-post-2019-11-07-1.md create mode 100644 docs/news/posts/blogger-post-2019-11-12-1.md create mode 100644 docs/news/posts/blogger-post-2020-01-24-1.md create mode 100644 docs/news/posts/blogger-post-2020-03-07-1.md create mode 100644 docs/news/posts/blogger-post-2020-03-10-1.md create mode 100644 docs/news/posts/blogger-post-2020-03-18-1.md create mode 100644 docs/news/posts/blogger-post-2020-04-03-1.md create mode 100644 docs/news/posts/blogger-post-2020-05-18-1.md create mode 100644 docs/news/posts/blogger-post-2020-07-04-1.md create mode 100644 docs/news/posts/blogger-post-2020-07-26-1.md create mode 100644 docs/news/posts/blogger-post-2020-09-17-1.md create mode 100644 docs/news/posts/blogger-post-2020-09-28-1.md create mode 100644 docs/news/posts/blogger-post-2020-10-03-1.md create mode 100644 docs/news/posts/blogger-post-2021-05-31-1.md create mode 100644 docs/news/posts/blogger-post-2021-06-24-1.md create mode 100644 docs/news/posts/blogger-post-2021-07-05-1.md create mode 100644 docs/news/posts/blogger-post-2021-09-23-1.md create mode 100644 docs/news/posts/blogger-post-2021-10-08-1.md create mode 100644 docs/news/posts/blogger-post-2021-10-08-2.md create mode 100644 docs/news/posts/blogger-post-2021-10-11-1.md create mode 100644 docs/news/posts/blogger-post-2021-10-23-1.md create mode 100644 docs/news/posts/blogger-post-2021-10-24-1.md create mode 100644 docs/news/posts/blogger-post-2021-11-02-1.md create mode 100644 docs/news/posts/blogger-post-2021-11-14-1.md create mode 100644 docs/news/posts/blogger-post-2021-11-18-1.md create mode 100644 docs/news/posts/blogger-post-2021-12-20-1.md create mode 100644 docs/news/posts/blogger-post-2021-12-20-2.md create mode 100644 docs/news/posts/blogger-post-2022-01-21-1.md create mode 100644 docs/news/posts/blogger-post-2022-02-05-1.md create mode 100644 docs/news/posts/blogger-post-2022-02-07-1.md create mode 100644 docs/news/posts/blogger-post-2022-02-08-1.md create mode 100644 docs/news/posts/blogger-post-2022-02-14-1.md create mode 100644 docs/news/posts/blogger-post-2022-03-11-1.md create mode 100644 docs/news/posts/blogger-post-2022-03-13-1.md create mode 100644 docs/news/posts/blogger-post-2022-03-15-1.md create mode 100644 docs/news/posts/blogger-post-2022-04-03-1.md create mode 100644 docs/news/posts/blogger-post-2022-05-13-1.md create mode 100644 docs/news/posts/blogger-post-2022-08-27-1.md create mode 100644 docs/news/posts/blogger-post-2022-09-11-1.md create mode 100644 docs/news/posts/blogger-post-2023-01-18-1.md create mode 100644 docs/news/posts/blogger-post-2024-02-09-1.md create mode 100644 docs/news/posts/blogger-post-2024-08-11-1.md create mode 100644 docs/news/posts/blogger-post-2024-08-11-2.md diff --git a/docs/news/posts/blogger-post-2006-09-23-1.md b/docs/news/posts/blogger-post-2006-09-23-1.md new file mode 100644 index 0000000..d98a1e5 --- /dev/null +++ b/docs/news/posts/blogger-post-2006-09-23-1.md @@ -0,0 +1,6 @@ +--- +date: 2006-09-23 +--- + +# Open Computing Facility Blog Launch +In an effort to improve communication with our users, the Open Computing Facility has launched a blog! Status and feature updates, random staff trivia, and other tasty bits of information will be posted here on a regular basis. This blog will also serve as a means for notifying users of unexpected service interruptions. diff --git a/docs/news/posts/blogger-post-2006-09-25-1.md b/docs/news/posts/blogger-post-2006-09-25-1.md new file mode 100644 index 0000000..2798446 --- /dev/null +++ b/docs/news/posts/blogger-post-2006-09-25-1.md @@ -0,0 +1,6 @@ +--- +date: 2006-09-25 +--- + +# Random Issues and Printer Information +The Open Computing Facility is experiencing some issues with our web server and some core services at the moment. We're working on figuring out what's broken (for the technically-minded, our NFS server seems to be borked) and getting it working again. I'll post status updates as they come.

Oh, and we've decided to order another maintenance kit for our printer. You might have noticed splotches appearing on printouts in the last few weeks -- our fuser was slowly dying, so we ordered a kit to fix it. Unfortunately, our order apparently got lost by CDW, so we're overnighting another kit. Our printer should be back soon! For now, we're using one of our older printers, so print times might be a bit long...

UPDATE #1: It's past 1 AM, and the OCF is still broken. sluo came in and is working diligently on fixing stuff because I don't know enough about NIS+ and NFS to do anything :/

UPDATE #2: Our super ex-SM was able to bring most services up by 2 AM. I think. Anyways, a bunch of other staffers showed up to provide moral support (by playing Quake 3). Here's some pictures of us staffers working for you -- we're always working for you (with regards to Verizon)...

OCF Tech Support Party at 2 AM

OCF Tech Support Party at 2 AM

Update #3: It's almost 3 AM, and we're still in here. Anybody know of any 24 hour food places that are within walking distance of Sproul or that deliver? diff --git a/docs/news/posts/blogger-post-2006-09-26-1.md b/docs/news/posts/blogger-post-2006-09-26-1.md new file mode 100644 index 0000000..4369277 --- /dev/null +++ b/docs/news/posts/blogger-post-2006-09-26-1.md @@ -0,0 +1,6 @@ +--- +date: 2006-09-26 +--- + +# Printer Fixed! +Our printer repair kit arrived this morning, and I installed it into our printer with tender loving care. Print jobs are now whizzing through the print queue, and there are no more splotches on the printouts (thanks to the new fuser)!

Now I just wonder how long this new printer happiness is going to last before the next breakdown... diff --git a/docs/news/posts/blogger-post-2006-09-29-1.md b/docs/news/posts/blogger-post-2006-09-29-1.md new file mode 100644 index 0000000..a6224b4 --- /dev/null +++ b/docs/news/posts/blogger-post-2006-09-29-1.md @@ -0,0 +1,6 @@ +--- +date: 2006-09-29 +--- + +# Penguins! Everywhere! +PEGUINS, PEGUINS IN THE LAB!!!
AHH PENGUINS EVERYWHERE!!!


-jlee

Callug penguin take over OCF on friday 2:30am diff --git a/docs/news/posts/blogger-post-2006-10-02-1.md b/docs/news/posts/blogger-post-2006-10-02-1.md new file mode 100644 index 0000000..982fe43 --- /dev/null +++ b/docs/news/posts/blogger-post-2006-10-02-1.md @@ -0,0 +1,6 @@ +--- +date: 2006-10-02 +--- + +# Shiny New Lab and Computers +We've cleaned up the OCF lab and installed a bunch of super fast new computers running Debian Linux. Please give the machines a shot; IMHO, they're much easier to use and more reliable than our Windows systems (this is not a dig at the Windows team, who have been practically living in the OCF to ensure that the systems work). diff --git a/docs/news/posts/blogger-post-2006-10-06-1.md b/docs/news/posts/blogger-post-2006-10-06-1.md new file mode 100644 index 0000000..4b0d66d --- /dev/null +++ b/docs/news/posts/blogger-post-2006-10-06-1.md @@ -0,0 +1,6 @@ +--- +date: 2006-10-06 +--- + +# We're Always Working for You! +Even in our sleep...

Jameson Hard at Work diff --git a/docs/news/posts/blogger-post-2006-10-12-1.md b/docs/news/posts/blogger-post-2006-10-12-1.md new file mode 100644 index 0000000..1e980df --- /dev/null +++ b/docs/news/posts/blogger-post-2006-10-12-1.md @@ -0,0 +1,6 @@ +--- +date: 2006-10-12 +--- + +# (More) Unscheduled Downtime +The OCF is currently (mostly) down due to a hardware failure on one of our core servers (specifically, war appears to have lost its SCSI controller). We're working to restore service; check back here for updates. diff --git a/docs/news/posts/blogger-post-2006-10-12-2.md b/docs/news/posts/blogger-post-2006-10-12-2.md new file mode 100644 index 0000000..0927a6f --- /dev/null +++ b/docs/news/posts/blogger-post-2006-10-12-2.md @@ -0,0 +1,6 @@ +--- +date: 2006-10-12 +--- + +# Status Update +We're still working on the problem. It seems like our disk array that holds all user data is having some troubles. I'm currently backing up all user mail to a safe location, and we're working on doing the same with home directories (ie., your regular data and web space). diff --git a/docs/news/posts/blogger-post-2006-10-12-3.md b/docs/news/posts/blogger-post-2006-10-12-3.md new file mode 100644 index 0000000..b88a01b --- /dev/null +++ b/docs/news/posts/blogger-post-2006-10-12-3.md @@ -0,0 +1,6 @@ +--- +date: 2006-10-12 +--- + +# Status Update +Mail is pretty much safe for the moment. We have two complete backups of mail stored on different systems. We're getting some errors while performing our initial backup of user data, but we're hoping that these errors are only temporary (ie., they'll be solved when we fsck the file system).

Some users have requested an ETA, and, for the moment, it seems the earliest the OCF will be back up and in working order is this weekend. No guarantees, though. Although we know how important it is for our users to have access to their mail, data, and web services, we're trying to take our time and do everything right so no user data is lost. If there are any special circumstances or issues that you believe we should be aware of, please visit our IRC channel (#ocf on irc.ocf.berkeley.edu). diff --git a/docs/news/posts/blogger-post-2006-10-13-1.md b/docs/news/posts/blogger-post-2006-10-13-1.md new file mode 100644 index 0000000..f12e607 --- /dev/null +++ b/docs/news/posts/blogger-post-2006-10-13-1.md @@ -0,0 +1,6 @@ +--- +date: 2006-10-13 +--- + +# Status Update +We're still backing up the rest of user data on our disk array to some spare space we have on our servers. Since we have upwards of 400 GB of data, and we're transferring most of it over NFS (regular Ethernet and not SCSI or Fibre Channel), it's taking a long time.

Some users have asked about data loss during this recovery. Most mail daemons should be smart enough to retry delivery once service to the OCF is restored. If our downtime ends up becoming prolonged, we will try to figure out a way to queue mail so it doesn't end up getting bounced.

In regards to user data (ie., anything other than mail), we're pulling the data off the disk array as quickly as possible. So far, it seems like most user data is intact; we're only getting about 1-2% corruption. That's not to say that that 1-2% of data is lost; we're just pulling the good data from the disk array at the moment. We haven't even begun to run the Unix equivalent of Scandisk, so it seems like there's a good chance we'll have 100% data recovery. Keep your fingers crossed, though.

Beyond the fact that we're working with such a massive amount of data, one of the holdups on our recovery is acquiring a LSI Logic PCI-X SAS/SATA host controller that supports Sun Solaris SPARC so we can setup a staging area to backup our disk array. If you don't understand what all those acronyms mean, let's just say you can't walk into any CompUSA or BestBuy and find that card. The only place that seems to carry the card is Newegg, but it's $300, and, even with overnight shipping, the earliest we're getting it is Monday.

Yury and I (the current site-managers) have been taking long shifts in the OCF to get user data back, and most of the other staff have been around to provide assistance (thanks sluo for saving us when we don't know Solaris 10!), so, we're working on it! diff --git a/docs/news/posts/blogger-post-2006-10-13-2.md b/docs/news/posts/blogger-post-2006-10-13-2.md new file mode 100644 index 0000000..e9629a8 --- /dev/null +++ b/docs/news/posts/blogger-post-2006-10-13-2.md @@ -0,0 +1,6 @@ +--- +date: 2006-10-13 +--- + +# Status Update +Some OCF staffers are making a trip to Fremont to pick up some 500 GB drives so we have more space to backup user data. They should be back in Berkeley by 5 PM today, and I'll be working through the night to setup the drives so that we can dump data to them.

Home directories have been successfully backed up, and we're currently going through web space. After web space, we'll have Microsoft Windows profiles and MySQL/Postgresql databases left to backup (I'm sure the other staffers will correct me if I'm missing something here).

Oh, and in regards to a user's comment, yes, mail in other directories should be restored (they're part of the home directories, which are almost done being dumped from the array).

Thanks for all your support! diff --git a/docs/news/posts/blogger-post-2006-10-14-1.md b/docs/news/posts/blogger-post-2006-10-14-1.md new file mode 100644 index 0000000..85d76f6 --- /dev/null +++ b/docs/news/posts/blogger-post-2006-10-14-1.md @@ -0,0 +1,6 @@ +--- +date: 2006-10-14 +--- + +# Status Update +We're almost done backing up all user data. While the backup was going on, we were able to assemble a simple 1 TB ZFS array using our newly acquired 500 GB Seagate SATA drives from Frys. Once the backup finishes, we'll do a raw dump of the disk array (where user data was stored) to our ZFS array (which we just built yesterday night). This will provide a secondary backup, just in case things go wrong -- we want to be extra careful with user data. Once that completes, we'll perform a fsck of the disk array, and, if everything goes well, most or all user data should be safe and accessible, and we'll start bringing OCF services back up. In other words, if everything does go well, some OCF services should be back up by the end of the weekend.

Now, if things go wrong, and the disk array starts spitting out errors, we're going to attempt to recover data from our ZFS array (it's basically our backup-backup). If that'll result in too much downtime, we'll dump our first backup onto the disk array (that is, the copy where we have 99% of user data or so) and work on bringing the OCF back up as quickly as possible. That way, users will have access to their data as soon as possible, and we'll work on restoring the extra 1% of data as we can from our secondary backup, without too much pressure on time. diff --git a/docs/news/posts/blogger-post-2006-10-15-1.md b/docs/news/posts/blogger-post-2006-10-15-1.md new file mode 100644 index 0000000..4a25c74 --- /dev/null +++ b/docs/news/posts/blogger-post-2006-10-15-1.md @@ -0,0 +1,6 @@ +--- +date: 2006-10-15 +--- + +# Status Update +The fsck didn't go so well. We're restoring our secondary backup of the disk array and going for another attempt at fixing the file system. This will be our final attempt at repairing the file system; we don't want to prolong our downtime since the process of restoring the backup to the disk array takes upwards of 10 hours. If we are unable to restore the file system, we'll wipe the disk array clean, create a new UFS file system, and rebuild user data from the tar archives we created on Friday and Saturday.

That is, our first attempt at repairing the file system failed. We're going to try again, but we're trying to balance our efforts at recovery with minimizing downtime. If we can't repair our file system, we're just going to wipe the slate clean and pull data from an archive we made, which may be missing a very small fraction of user data (basically the data that was damaged during the initial hardware failure). Our worst case estimate is around 1% data loss; most users won't be affected, and for the users that were, most files that we were unable to be recover seem to be unimportant files (browser cache files, temporary lock files, etc).

So, just to be clear: we're trying our best to get 100% data recovery, but doing so while minimizing downtime is difficult. Our worst case scenario is bringing back the OCF with about 99% of the data intact and working with users to recover any important data from the 1% that may be lost. diff --git a/docs/news/posts/blogger-post-2006-10-16-1.md b/docs/news/posts/blogger-post-2006-10-16-1.md new file mode 100644 index 0000000..1129b92 --- /dev/null +++ b/docs/news/posts/blogger-post-2006-10-16-1.md @@ -0,0 +1,6 @@ +--- +date: 2006-10-16 +--- + +# Status Update +I'm about to head back to the OCF and swap our SCSI connectors to our disk array so we can continue with our fsck recovery efforts. In the mean time, the other staffers are working on bringing our mail server back online so we can queue incoming mail. diff --git a/docs/news/posts/blogger-post-2006-10-16-2.md b/docs/news/posts/blogger-post-2006-10-16-2.md new file mode 100644 index 0000000..6e8a7d3 --- /dev/null +++ b/docs/news/posts/blogger-post-2006-10-16-2.md @@ -0,0 +1,6 @@ +--- +date: 2006-10-16 +--- + +# Status Update +A staffer requested that I create a new post for every status update, as it'll be much easier for people using RSS readers to stay informed, so I've taken all the previous status updates and created a new post for each one and will continue doing so in the future.

Our second attempt at repairing the file system failed. Therefore, per the course of action I mentioned in my last status update, we've decided to wipe the disk array clean and rebuild the file system from our primary backup. This should help minimize downtime and get the OCF back to peak performance ASAP.

Once the dust settles, we (the Site Managers) will probably be sending out a more formal email message describing the failure, our response to it, and how we plan on avoiding such failures in the future.

Oh, and I should note that we're currently queuing incoming mail. diff --git a/docs/news/posts/blogger-post-2006-10-16-3.md b/docs/news/posts/blogger-post-2006-10-16-3.md new file mode 100644 index 0000000..91e0f5c --- /dev/null +++ b/docs/news/posts/blogger-post-2006-10-16-3.md @@ -0,0 +1,6 @@ +--- +date: 2006-10-16 +--- + +# Status Update +A new file system was created on the disk array around 5 AM this morning, and we've been transferring data back to the array over NFS and regular Ethernet (silly endian issues prevent us from connecting the array directly to our backup system using U320 SCSI). At the current rate, we should be finished transferring the files over sometime tomorrow. A staffer might be able to drop by the lab today and figure out a better wiring method to improve transfer speeds and to get data on the array faster.

Also, here's a lesson for all system administrators out there: DO NOT BUY DISK ARRAYS FROM SHADY VENDORS. Since our budget is relatively limited, we've always been pretty conservative with our purchases and primarily relied upon donations to keep us going (thanks to Sun Microsystems for our super fast servers!). Consequently, when we needed to expand the OCF's disk offerings, we were only able to justify the purchase of a 'budget' disk array. That was 2 years ago. Our disk array is currently failing, and the company we bought the disk array from went out of business and was bought by some other company, who only wants to perform service via RMA through a process that might take 15 days.

ARE THEY FREAKING INSANE? So, we're supposed to send them our 3U form-factor disk array with 12 drives via postal mail and be down for 15 more days? Uh, no thanks.

Ok, the end of my status update and rant. diff --git a/docs/news/posts/blogger-post-2006-10-17-1.md b/docs/news/posts/blogger-post-2006-10-17-1.md new file mode 100644 index 0000000..a524735 --- /dev/null +++ b/docs/news/posts/blogger-post-2006-10-17-1.md @@ -0,0 +1,6 @@ +--- +date: 2006-10-17 +--- + +# Status Update +Files are still being copied to the array, but at the current rate, we should definitely have all files back on our disk array by Tuesday morning. If all goes well, the OCF should be back up and running by Tuesday night or Wednesday morning. I hope.

Since we're already down, we've decided to migrate our mail service over to a much faster server. Hopefully that'll allow some good to come out of this entire mess... diff --git a/docs/news/posts/blogger-post-2006-10-17-2.md b/docs/news/posts/blogger-post-2006-10-17-2.md new file mode 100644 index 0000000..cdc1e56 --- /dev/null +++ b/docs/news/posts/blogger-post-2006-10-17-2.md @@ -0,0 +1,6 @@ +--- +date: 2006-10-17 +--- + +# Status Update +All user data has been restored to the disk array. I'm currently running a fsck on the array just to make sure that the array hasn't already corrupted it. We'll be keeping very regular backups of user data until we can figure out what's wrong with our disk array or until we can get it serviced, so there shouldn't be any future extended downtime like this again (at least as a result of the disk array).

We're on track for the OCF coming back online sometime tonight or tomorrow morning. As an added safety precaution, I'm currently setting up a two-way RAID-1 mirror with hot-spare on our primary NFS server (basically the computer that serves all user files) to make everything triply redundant.

Thanks for all your support through this process! diff --git a/docs/news/posts/blogger-post-2006-10-18-1.md b/docs/news/posts/blogger-post-2006-10-18-1.md new file mode 100644 index 0000000..8a3f069 --- /dev/null +++ b/docs/news/posts/blogger-post-2006-10-18-1.md @@ -0,0 +1,6 @@ +--- +date: 2006-10-18 +--- + +# OCF Sorta Back Up +We've just finished restoring most OCF services. Home directories and web pages should work, and logins to all our general servers should also work. There might be some glitches here and there as we put the finishing touches on our restoration, though, so please bear with us.

Mail is still offline, but we're still queuing mail. The main reason mail remains offline is that we need to run all the mail we've queued up through our mail delivery system again, and we can only really do that once we're sure everything works. diff --git a/docs/news/posts/blogger-post-2006-10-19-1.md b/docs/news/posts/blogger-post-2006-10-19-1.md new file mode 100644 index 0000000..352a3cc --- /dev/null +++ b/docs/news/posts/blogger-post-2006-10-19-1.md @@ -0,0 +1,6 @@ +--- +date: 2006-10-19 +--- + +# More Services Being Restored +We're re-enabled access to mail that was delivered before our array failure on Thursday. Users should be able to read and manage their old mail. Unfortunately, you'll need to login to one of our shell servers to read your mail; we haven't brought POP or IMAP back up yet. At the same time, we're working on delivering mail that was queued during the downtime. Please have patience; we're working very hard on it.

Also, we're aware of the issues with databases and are trying to debug the problem.

I'm currently attempting to probe our disk image of the array to see if I can find any files that were lost during the restoration process. Since the image is about ~1 TB, it's not going as quickly as I'd like, but it's still running. diff --git a/docs/news/posts/blogger-post-2006-10-22-1.md b/docs/news/posts/blogger-post-2006-10-22-1.md new file mode 100644 index 0000000..654f675 --- /dev/null +++ b/docs/news/posts/blogger-post-2006-10-22-1.md @@ -0,0 +1,6 @@ +--- +date: 2006-10-22 +--- + +# Mail and Database Status Update +We've begun delivering mail that was queued up during our array failure outage. We've also enabled IMAP and POP3 access again, so users should be able to read their mail using their favorite mail client. It may take a couple days for all mail to be delivered; hundred of thousands of messages were queued up during the outage.

MySQL databases have been restored as well as we can restore them. Once we re-import all the databases into our MySQL server, we'll bring MySQL back online. A small number of users have irrecoverable errors with their databases; we will be contacting each of the affected users individually to work with them on recovering their data. diff --git a/docs/news/posts/blogger-post-2006-10-26-1.md b/docs/news/posts/blogger-post-2006-10-26-1.md new file mode 100644 index 0000000..16e5cb3 --- /dev/null +++ b/docs/news/posts/blogger-post-2006-10-26-1.md @@ -0,0 +1,6 @@ +--- +date: 2006-10-26 +--- + +# Random Outages +Over the course of the past day and a half, we have been experiencing some random errors with our primary authentication system. These errors have led to some difficulties logging in for some users and some other problems in the physical lab (printer queue jams, frozen terminals, etc). We're not quite sure of what's causing the problems, but we have a pretty good idea it's related to us using NIS+ (an old standard developed by Sun Microsystems that has been deprecated). Thanks to sluo, we were able to recover from these errors, so everything should be up and working now.

In regards to our other services:

The mail queue is still being processed, but there's still a huge chunk of mail that's left in the queue.

MySQL databases should be restored as well as we can restore them. Users with data that we have identified as problematic in recovering will be individually contacted via email tomorrow evening (I'm consolidating a list of the errors we received so I can send it all in one pass).

PostgreSQL is currently being looked at and debugged.

We've found a way to get our disk array serviced, but it means sending back critical parts of our disk array. Since downtime is unacceptable, we're going to build a temporary disk array out of commodity parts and use that in a 'hot-swap' manner with our current disk array. I'm waiting for the parts in the mail, though...

Sorry about not updating this blog in the past couple days, but I've been rather busy, and I only got to leave the OCF around 4 AM yesterday morning. diff --git a/docs/news/posts/blogger-post-2006-10-28-1.md b/docs/news/posts/blogger-post-2006-10-28-1.md new file mode 100644 index 0000000..8effe14 --- /dev/null +++ b/docs/news/posts/blogger-post-2006-10-28-1.md @@ -0,0 +1,6 @@ +--- +date: 2006-10-28 +--- + +# Newegg's Automation Needs Work +On Sunday, we ordered another 500 GB SATA2 drive from Newegg along with a LSI Logic SAS/SATA HBA so we could build a temporary array while we had our current array serviced. The plan was to move all OCF data to this 'new' array temporarily while we sent our currently broken array back to the manufacturer. So, our Newegg package finally arrived today.

Except there was one slight problem: Newegg sent us a PATA drive instead of a SATA drive. So, now we have three SATA drives and one PATA drive. You put the four things together, and you don't have a SATA array.

Newegg is closed for the weekend, so we're trying to see if we can acquire a SATA drive over the weekend and get to building our temporary disk array.

This past month has got to be an example of Murphy's Law in action. diff --git a/docs/news/posts/blogger-post-2006-10-29-1.md b/docs/news/posts/blogger-post-2006-10-29-1.md new file mode 100644 index 0000000..f69a408 --- /dev/null +++ b/docs/news/posts/blogger-post-2006-10-29-1.md @@ -0,0 +1,6 @@ +--- +date: 2006-10-29 +--- + +# makehttp works again +I just fixed makehttp. Users should be able to create their own web directories once again.

For those that didn't already know, OCF accounts come with your own web space. You can activate it by executing 'makehttp' at a command prompt. You can store files that you want to be accessible to the Internet in your 'public_html' directory, and the URL to your web space will be www.ocf.berkeley.edu/~username. diff --git a/docs/news/posts/blogger-post-2006-11-05-1.md b/docs/news/posts/blogger-post-2006-11-05-1.md new file mode 100644 index 0000000..e18dba6 --- /dev/null +++ b/docs/news/posts/blogger-post-2006-11-05-1.md @@ -0,0 +1,6 @@ +--- +date: 2006-11-05 +--- + +# Possible Downtime on Monday +We have been alerted to a failing memory module on our web, database, and printing server and have scheduled an on-site visit by a Sun Microsystems field engineer to replace the defective memory module. Assuming that everything goes to plan, there will be a slight interruption in our web, database, and in-lab printing services around noon on Monday, November 6 as we repair our server. We are anticipating a maximum downtime of two hours.

No other OCF services should be affected (ex., mail and shell accounts). If the repair takes less than two hours, I'll use the remaining time to upgrade the system software on our server. If the repair process extends beyond the allocated time, status updates will be posted to this blog. diff --git a/docs/news/posts/blogger-post-2006-11-07-1.md b/docs/news/posts/blogger-post-2006-11-07-1.md new file mode 100644 index 0000000..ce561c9 --- /dev/null +++ b/docs/news/posts/blogger-post-2006-11-07-1.md @@ -0,0 +1,6 @@ +--- +date: 2006-11-07 +--- + +# Server Downtime Postponed +The Sun Microsystems field engineer called today and told me that the replacement hardware for our server was on backorder. We'll probably be rescheduling the server downtime for sometime this week or early next week. diff --git a/docs/news/posts/blogger-post-2006-11-11-1.md b/docs/news/posts/blogger-post-2006-11-11-1.md new file mode 100644 index 0000000..e44568e --- /dev/null +++ b/docs/news/posts/blogger-post-2006-11-11-1.md @@ -0,0 +1,6 @@ +--- +date: 2006-11-11 +--- + +# Disk Array Migration Ongoing +We're currently migrating the OCF off our current disk array onto a temporary disk array to allow our regular disk array to be serviced. According to the OCF announcement that was sent out yesterday, we expect downtime to end at approximately 8 AM Saturday morning (November 11). diff --git a/docs/news/posts/blogger-post-2006-11-11-2.md b/docs/news/posts/blogger-post-2006-11-11-2.md new file mode 100644 index 0000000..2c6d014 --- /dev/null +++ b/docs/news/posts/blogger-post-2006-11-11-2.md @@ -0,0 +1,6 @@ +--- +date: 2006-11-11 +--- + +# Status Update +Files are still being copied, it's raining in Berkeley, and we might slip past our target deadline of 8 AM by a few hours.

I'll post more information as it comes. diff --git a/docs/news/posts/blogger-post-2006-11-15-1.md b/docs/news/posts/blogger-post-2006-11-15-1.md new file mode 100644 index 0000000..7e5eb63 --- /dev/null +++ b/docs/news/posts/blogger-post-2006-11-15-1.md @@ -0,0 +1,6 @@ +--- +date: 2006-11-15 +--- + +# Bumpy Services +I apologize for the lack of status updates in the past few days. A lot of things have been occurring inside and outside of the OCF, so I haven't been able to dedicate a large quantity of my time towards updating this blog. Here's a quick summary of our disk array migration:


Now that we've transferred our services off of our primary disk array, I'm going to be working on restoring an old image of our disk array so that we can restore files that were overlooked during our disk array failure last month.

Thanks for your patience. diff --git a/docs/news/posts/blogger-post-2006-11-16-1.md b/docs/news/posts/blogger-post-2006-11-16-1.md new file mode 100644 index 0000000..80e1b2b --- /dev/null +++ b/docs/news/posts/blogger-post-2006-11-16-1.md @@ -0,0 +1,6 @@ +--- +date: 2006-11-16 +--- + +# Accounts Being Created +Account creation has been restarted. If you requested an account within the past month, it should have been created last night. diff --git a/docs/news/posts/blogger-post-2006-11-20-1.md b/docs/news/posts/blogger-post-2006-11-20-1.md new file mode 100644 index 0000000..5c4c8e9 --- /dev/null +++ b/docs/news/posts/blogger-post-2006-11-20-1.md @@ -0,0 +1,6 @@ +--- +date: 2006-11-20 +--- + +# Temporary Array Fixed +The lockup issues we were having with our temporary disk array have been resolved after a BIOS and firmware update were applied to our SAS HBA.

Also, please do not reboot our Linux computers, even if they are appear to be hung-up. Numerous reboots damage the hardware (especially the hard drive and power supply), and since those items are costly to replace, we end up having to take the machines offline for awhile. diff --git a/docs/news/posts/blogger-post-2006-11-23-1.md b/docs/news/posts/blogger-post-2006-11-23-1.md new file mode 100644 index 0000000..2dd098e --- /dev/null +++ b/docs/news/posts/blogger-post-2006-11-23-1.md @@ -0,0 +1,6 @@ +--- +date: 2006-11-23 +--- + +# Happy Thanksgiving! +We're hoping you're having a nice breather from classes and spending time with your loved ones. diff --git a/docs/news/posts/blogger-post-2006-11-29-1.md b/docs/news/posts/blogger-post-2006-11-29-1.md new file mode 100644 index 0000000..248d10b --- /dev/null +++ b/docs/news/posts/blogger-post-2006-11-29-1.md @@ -0,0 +1,6 @@ +--- +date: 2006-11-29 +--- + +# PHP5 Beta Test +PHP5 has been setup on our web server. Any files with a '.php5' extension should be run with the PHP5 interpreter.

If you notice any bugs, please contact staff or post a comment. diff --git a/docs/news/posts/blogger-post-2006-11-30-1.md b/docs/news/posts/blogger-post-2006-11-30-1.md new file mode 100644 index 0000000..4e20747 --- /dev/null +++ b/docs/news/posts/blogger-post-2006-11-30-1.md @@ -0,0 +1,6 @@ +--- +date: 2006-11-30 +--- + +# Home Directories Restored +All home directories that were missing after the disk array failure have been restored. diff --git a/docs/news/posts/blogger-post-2006-12-10-1.md b/docs/news/posts/blogger-post-2006-12-10-1.md new file mode 100644 index 0000000..8219794 --- /dev/null +++ b/docs/news/posts/blogger-post-2006-12-10-1.md @@ -0,0 +1,6 @@ +--- +date: 2006-12-10 +--- + +# New Linux Login Server +The OCF has a new Linux login server thanks to a recent hardware donation. This server is configured with the same software as our AMD64 lab machines but is secured in our server room, so it should be more reliable and less-prone to random reboots.

Just log into tsunami.ocf.berkeley.edu with your OCF account to use this new server. diff --git a/docs/news/posts/blogger-post-2007-02-04-1.md b/docs/news/posts/blogger-post-2007-02-04-1.md new file mode 100644 index 0000000..9d42663 --- /dev/null +++ b/docs/news/posts/blogger-post-2007-02-04-1.md @@ -0,0 +1,6 @@ +--- +date: 2007-02-04 +--- + +# Web Application Testing Machine +Due to popular demand, we've created a Solaris zone on the hardware that hosts our web server so that users can directly test their web applications.

Just login to asteroid.ocf.berkeley.edu and get your web application development going! diff --git a/docs/news/posts/blogger-post-2007-02-04-2.md b/docs/news/posts/blogger-post-2007-02-04-2.md new file mode 100644 index 0000000..26902d3 --- /dev/null +++ b/docs/news/posts/blogger-post-2007-02-04-2.md @@ -0,0 +1,6 @@ +--- +date: 2007-02-04 +--- + +# Unhappy Web Server +As many of you have noticed, our web server has been experiencing lots of uptime issues. We're not exactly sure what's causing the problem, but we think it may be related to NFS and some prior issues we've had with the Solaris install on the machine. For now, it seems that rebooting the web server fixes the problem for a few hours, so, if you notice any issues with our web services, contact staff and we'll do something.

In regards to a long term solution to these problems, we are planning an overhaul of the Solaris install on the machine (read: re-install of the operating system). Unfortunately, this process is complicated by the fact that we have many other services hosted on the machine (in-lab printing, databases, etc), so check here for status updates regarding our repair plans. diff --git a/docs/news/posts/blogger-post-2007-02-05-1.md b/docs/news/posts/blogger-post-2007-02-05-1.md new file mode 100644 index 0000000..c6a8358 --- /dev/null +++ b/docs/news/posts/blogger-post-2007-02-05-1.md @@ -0,0 +1,6 @@ +--- +date: 2007-02-05 +--- + +# OCF Website Refresh +We are currently rolling out a facelifted look for the OCF pages, with XHTML1.0 Transitional and CSS compliant source code under the hood, starting with the most static pages and working the way towards the more heavily dynamic pages.

We're testing vigorously across the Firefox, Opera, Safari, and IE browsers, but there will be a few bugs along the way. So feel free to send any feedback our way. We're listening. Enjoy!

diff --git a/docs/news/posts/blogger-post-2007-02-26-1.md b/docs/news/posts/blogger-post-2007-02-26-1.md new file mode 100644 index 0000000..de1f8eb --- /dev/null +++ b/docs/news/posts/blogger-post-2007-02-26-1.md @@ -0,0 +1,6 @@ +--- +date: 2007-02-26 +--- + +# Oops +Some of you may have noticed that we had prolonged downtime on Wednesday, Feb 21. Long story short, I screwed up.

I was trying to fix our continuing file server problems by making changes to our server configuration and applying some Solaris patches. I brought the server and our core services down for a reboot, applied some patches, and booted up the system. Unfortunately, the boot process hung, so I waited a few minutes and tried rebooting. No dice.

After a few cycles of rebooting, monitoring, and various attempts at server black magic, I had pretty much given up when sluo, a former OCF Site Manager signed on and helped bring the server backup.

We're pretty sure a Solaris patch was to blame. The patch had worked on other OCF machines, but apparently our bleeding-edge SunFire T2000 didn't like it. diff --git a/docs/news/posts/blogger-post-2007-03-04-1.md b/docs/news/posts/blogger-post-2007-03-04-1.md new file mode 100644 index 0000000..4a0571d --- /dev/null +++ b/docs/news/posts/blogger-post-2007-03-04-1.md @@ -0,0 +1,6 @@ +--- +date: 2007-03-04 +--- + +# Web and Database Server Upgrade +We are currently upgrading the operating system on our web and database host. I don't have an ETA, but we're doing our best to complete the process as quickly as possible. Thanks for your patience! diff --git a/docs/news/posts/blogger-post-2007-03-06-1.md b/docs/news/posts/blogger-post-2007-03-06-1.md new file mode 100644 index 0000000..6cebae5 --- /dev/null +++ b/docs/news/posts/blogger-post-2007-03-06-1.md @@ -0,0 +1,6 @@ +--- +date: 2007-03-06 +--- + +# Web Services (Mostly) Restored +Most of our web services have been restored.

We had to merge our web and database servers into our new Solaris install by hand, so a small number of Perl/Python/PHP modules and various binaries may have been overlooked and forgotten. If you think we may have missed something, do not hesitate to let staff know! We'll get right on it!

Again, I apologize for the prolonged downtime. We're hoping that this latest version of Solaris will prove to be more reliable than the last, resulting in greater reliability and performance for all OCF web users. diff --git a/docs/news/posts/blogger-post-2007-03-11-1.md b/docs/news/posts/blogger-post-2007-03-11-1.md new file mode 100644 index 0000000..95581cc --- /dev/null +++ b/docs/news/posts/blogger-post-2007-03-11-1.md @@ -0,0 +1,6 @@ +--- +date: 2007-03-11 +--- + +# State of the OCF Address +Brief History, Current State, and Future Plans

Executive Summary: Downtime in the near future to migrate disk array. Check back for exact date/times.

Some users have voiced concerns about being kept out of the loop regarding the state and future plans of OCF. Hopefully, this should bring everybody up to speed on what's been happening and what we have planned.

Many of our problems seem to be caused by the experimental hardware used on the server holding our backup disk array. Now that the primary array is ready to be brought back into service, we are expecting that a move back should fix many of our problems (mainly the webserver uptime and printing problems). After much consideration, we determined it would be best to connect the disk array to the machine that runs the webserver (famine), in part because this would eliminate a large amount of network traffic.

Before migrating data to the primary disk array, we decided it was best to make sure famine was completely up to date first. It had been running an older version of Solaris 10, and the latest update contained a a lot of bug fixes and security fixes. famine (through the use of virtual servers) runs our webserver, print server, and database server. This update was going to happen, and downtime was needed. By doing the upgrade first, users could still access our servers and their data during the downtime.

While it is possible to update the version that was already installed, we found that it would be better to use a clean install of the latest version. According to our research, it seems that the normal upgrade path often result in unexpected errors with the virtual servers (called Containers or Zones). The general recommendation is to backup the Containers and perform a clean install. The process boiled down to five major steps:
1. back up zone data and configuration files (for the 8 installed zones)
2. install Solaris 10 Update 3 and set up raid devices
3. install and configure new zones
4. manually merge data from zone backups into new zones
5. restore services

This process takes a long time to complete, so we scheduled this upgrade for a weekend (3/2-3/4). Unfortunately, it seems that insufficient warning was given about this downtime. I apologize for the insufficient warning and any inconveniences this many have caused. We will work on getting downtime warnings out earlier.

While we had originally planned to migrate the disk array data at the end of this upgrade process, we ran into some problems. During the upgrade, we started getting hardware fault errors concerning one of the CPUs and memory modules. At that point, we decided it would be best to hold off on the disk array migration until after the hardware issues had been resolved, so we finished with the upgrade and restored services.

After many days of troubleshooting and several on-site SUN service calls, we determined the problems were caused by a faulty memory controller on the mainboard. The mainboard has been replaced and it looks like we are ready to move ahead with the disk array migration. Since most of the setup has already been done, this is not too difficult a task, but it will require another extended downtime. The process boils down to:
1. make a recent sync of all data
2. kick all users off, shutdown web server
3. make final update of any changed data
4. reconfigure servers
5. bring everything back up

The final sync of user data must be performed when nobody can access the data so we don't get inconsistencies in the copies. This will require all users to be logged off and the webserver shutdown (i.e. downtime) sometime in the near future. This process will probably take something like 10 hours or so, depending on how much data changes between rsyncs. After the data is synced and the servers are reconfigured, we can bring everything back up and everybody should be happy.

Please post comments or email staff@ocf.berkeley.edu if you have any questions or concerns. diff --git a/docs/news/posts/blogger-post-2007-03-24-1.md b/docs/news/posts/blogger-post-2007-03-24-1.md new file mode 100644 index 0000000..e3c9d3b --- /dev/null +++ b/docs/news/posts/blogger-post-2007-03-24-1.md @@ -0,0 +1,6 @@ +--- +date: 2007-03-24 +--- + +# Downtime Update +We finished synchronizing user data between disk arrays overnight; we still need to sync our collection of installed software and some administrative files. That (hopefully, provided nothing goes wrong) puts us on track to restore service this afternoon or evening.

We're also taking advantage of this downtime to do some upgrading on one of our backend machines; that work has been completed. We might do some small physical reorganization of our servers and their power connections, but that's about it for the other work we're doing during this maintenance window.

Hope you're enjoying your break, and thanks for your patience! diff --git a/docs/news/posts/blogger-post-2007-03-25-1.md b/docs/news/posts/blogger-post-2007-03-25-1.md new file mode 100644 index 0000000..aa00b53 --- /dev/null +++ b/docs/news/posts/blogger-post-2007-03-25-1.md @@ -0,0 +1,6 @@ +--- +date: 2007-03-25 +--- + +# Disk Array Migration Complete +Ok, so we missed the target I posted here by a few hours (I forgot to account for staffers needing to travel home for the break), but we should be back up. Please note that quotas are once again being enforced; see our email from last week if you forget what this means. You have 30 days to get back under quota; those of you who are over quota should each receive mail about this sometime in the next few days.

There was a mistake in that email, by the way; the primary (soft) mail quota will stay at 10 MB for now, not go up to 25 MB as we announced. We're not necessarily short of disk space if we raise mail quotas (though I'm not sure if anyone's run the numbers), but our system stores mail quite inefficiently at the moment, which means that 25 MB inboxes would be pretty slow to access. Recall that if you need more space for mail, you can save it to another folder (which counts under the much larger home quota). Sorry!

We didn't have a copy of our old (pre-disk-array-failure) quotas handy, so everyone has the default soft quotas set at the moment. We'll see if we can find a copy of the old quotas in our backups, but in the meantime, if you previously obtained a quota increase, you may want to email us to remind us about it. Hard quotas were set such that no one would be over hard quota (though if your hard quota was raised because of this, don't expect it to stay that way forever), so you have 30 days to either clean up or remind us about your quota increase. (There were a few exceptions, for people who clearly were abusing disk usage.)

Again, enjoy your break! diff --git a/docs/news/posts/blogger-post-2007-03-29-1.md b/docs/news/posts/blogger-post-2007-03-29-1.md new file mode 100644 index 0000000..eb582b4 --- /dev/null +++ b/docs/news/posts/blogger-post-2007-03-29-1.md @@ -0,0 +1,6 @@ +--- +date: 2007-03-29 +--- + +# Website Refresh 1.5 +Both official OCF skins - "Tux" and "Arctic" - have been visually updated, primarily in the headers for improved color matching. Since February, a number of detail features have been implemented, including "Add search engine" autodiscovery from the Firefox 2 search bar based on OpenSearch (supported by Firefox 2 and Internet Explorer 7).

Another feature in the works is a web-based tool for uploading files to your website space (folder public_html), courtesy of Jonathan Chu. It's feature complete, and we hope to roll it out sometime soon. There are also plans to look into a web-based text editor for editing the source of your text-based files in your webspace, but we'll have news about that as things progress.

As always, if you have any suggestions, criticism, or pick-up lines, feel free to send them our way! Ciao.


Edit: You can try out the website file uploader, naturally at your own risk.


diff --git a/docs/news/posts/blogger-post-2007-05-02-1.md b/docs/news/posts/blogger-post-2007-05-02-1.md new file mode 100644 index 0000000..ffe6bf5 --- /dev/null +++ b/docs/news/posts/blogger-post-2007-05-02-1.md @@ -0,0 +1,6 @@ +--- +date: 2007-05-02 +--- + +# Account home directories erased +Accounts starting with "a" to "b[a-l]" have had their home directories accidentally erased and subsequently restored from last month's backup. It's complex. diff --git a/docs/news/posts/blogger-post-2007-05-20-1.md b/docs/news/posts/blogger-post-2007-05-20-1.md new file mode 100644 index 0000000..b4273a0 --- /dev/null +++ b/docs/news/posts/blogger-post-2007-05-20-1.md @@ -0,0 +1,6 @@ +--- +date: 2007-05-20 +--- + +# Sun Studio Compilers +We've updated the compilers available on our Solaris systems from Sun Forte 6 to Sun Studio 11. Enjoy! diff --git a/docs/news/posts/blogger-post-2007-06-01-1.md b/docs/news/posts/blogger-post-2007-06-01-1.md new file mode 100644 index 0000000..cd0bcca --- /dev/null +++ b/docs/news/posts/blogger-post-2007-06-01-1.md @@ -0,0 +1,6 @@ +--- +date: 2007-06-01 +--- + +# Server Reboots on Wednesday +As announced on our website and login server MOTDs, we rebooted one of our core servers on Wednesday for operating system maintenance. Downtime was as long as anticipated, but we also ended up having to reboot our login servers (conquest, apocalypse, and tsunami) since they were having issues. Everything's working now, and we thank you for your continued patience as we make changes to our infrastructure to deliver more reliable services to all OCF users.

We also have some really neat features in development this summer. One of the most anticipated projects is a self-serve backup system, which will allow OCF users to access daily snapshots of their data for at least the past seven days. More information will be posted as we approach a public beta test. diff --git a/docs/news/posts/blogger-post-2007-06-15-1.md b/docs/news/posts/blogger-post-2007-06-15-1.md new file mode 100644 index 0000000..b9d24dd --- /dev/null +++ b/docs/news/posts/blogger-post-2007-06-15-1.md @@ -0,0 +1,6 @@ +--- +date: 2007-06-15 +--- + +# Secure POP and IMAP transition +As of August 1, 2007, the OCF will be requiring all users to use secure connections to read their OCF email. Users who use webmail and/or local mail access (using a program such as pine, elm, or mutt from an OCF Unix machine) exclusively will not be affected. Otherwise, you will need to reconfigure your mail client to use secure connections when receiving mail; we have instructions on our wiki for some popular mail clients. (If you have a correction or clarification for us, or instructions for another client, please email them to staff@ocf.berkeley.edu.)

The latest information on the changes can be found on our wiki at http://docs.ocf.berkeley.edu/wiki/Major_service_changes.

If you have any questions or concerns, please don't hesitate to contact us (email staff@ocf.berkeley.edu). diff --git a/docs/news/posts/blogger-post-2007-06-19-1.md b/docs/news/posts/blogger-post-2007-06-19-1.md new file mode 100644 index 0000000..a275b30 --- /dev/null +++ b/docs/news/posts/blogger-post-2007-06-19-1.md @@ -0,0 +1,6 @@ +--- +date: 2007-06-19 +--- + +# Web Security Upgrade +One of the features that the OCF is rolling out this summer is increased web server security for our users and their web applications. We've added some tools that'll monitor web traffic and intercept many common types of attacks, giving our users an extra level of protection from newly discovered vulnerabilities.

Most importantly, though, these tools will help protect our users from comment spam (unsolicited commercial postings). The OCF web server receives more than one million requests every day, and a growing number of these requests are attempts by automated programs to post spam on our users' blogs, galleries, and web sites. Besides leaving distasteful messages, these programs also place a great burden upon our infrastructure, slowing down our services.

We've currently enabled an aggressive set of filters to catch most attacks and spam, and we will continue to add filters as new attacks are discovered. As with any automated monitoring system, though, there will always be the possibility for incorrectly tagged requests. If you're having problems with your web site, please let us know.

Also, please keep in mind that these security systems are designed to help us help you. We encourage OCF users to keep their web applications up-to-date, as bugs and security holes are being fixed every day, and it's impossible for us to protect every single application that our diverse user base employs. We also highly encourage users to install plugins to reduce the flow of spam via captchas or other methods. diff --git a/docs/news/posts/blogger-post-2007-06-30-1.md b/docs/news/posts/blogger-post-2007-06-30-1.md new file mode 100644 index 0000000..7689942 --- /dev/null +++ b/docs/news/posts/blogger-post-2007-06-30-1.md @@ -0,0 +1,6 @@ +--- +date: 2007-06-30 +--- + +# Upcoming Downtime +The OCF will be going down for maintenance on Sunday, July 1 for approximately 2 hours beginning at 6 PM. Mail will be queued up during the downtime and delivered when service resumes. diff --git a/docs/news/posts/blogger-post-2007-07-02-1.md b/docs/news/posts/blogger-post-2007-07-02-1.md new file mode 100644 index 0000000..caae9a1 --- /dev/null +++ b/docs/news/posts/blogger-post-2007-07-02-1.md @@ -0,0 +1,6 @@ +--- +date: 2007-07-02 +--- + +# System Status 2.0 +
The "System Status" page, linked from the front page, has been updated with improved accuracy of which OCF computers are on, as well as details such as uptime, names, and user numbers. (Thanks, Aaron). In addition to that, the graphics have been entirely redone to make them more recognizable. We hope you find it that much more useful, and don't forget that the lab webcam is broadcasting again. diff --git a/docs/news/posts/blogger-post-2007-08-01-1.md b/docs/news/posts/blogger-post-2007-08-01-1.md new file mode 100644 index 0000000..43fd44c --- /dev/null +++ b/docs/news/posts/blogger-post-2007-08-01-1.md @@ -0,0 +1,6 @@ +--- +date: 2007-08-01 +--- + +# Reminder: secure POP and IMAP now required +As previously announced, secure connections are now required to read your OCF email; in particular, POP and IMAP users who have not already configured their mail clients to use secure connections need to do so in order to keep using them. If you're not sure how to do this, have a look at our instructions for several popular mail clients.

If you have any questions or concerns, please don't hesitate to contact us (email staff@ocf.berkeley.edu). diff --git a/docs/news/posts/blogger-post-2007-08-04-1.md b/docs/news/posts/blogger-post-2007-08-04-1.md new file mode 100644 index 0000000..c37b116 --- /dev/null +++ b/docs/news/posts/blogger-post-2007-08-04-1.md @@ -0,0 +1,6 @@ +--- +date: 2007-08-04 +--- + +# Mail quota reporting in webmail +Following suggestions from staffers and others, our webmail interface now reports how much disk space your inbox is taking and how close you are to the maximum at the left of the screen, below the folders listing.

Reminder: unless you've received a quota increase, you're permitted to have 10 MB of mail in your inbox. Mail in folders other than the inbox does not count against your mail quota; it's recorded under your home directory quota (which is 400 MB by default). For more information, see our documentation. diff --git a/docs/news/posts/blogger-post-2007-09-07-1.md b/docs/news/posts/blogger-post-2007-09-07-1.md new file mode 100644 index 0000000..5df7898 --- /dev/null +++ b/docs/news/posts/blogger-post-2007-09-07-1.md @@ -0,0 +1,6 @@ +--- +date: 2007-09-07 +--- + +# First General Meeting for Fall 2007 +Welcome back OCFers!

We hope everyone is excited for a new school year with OCF.

Our first general meeting will be held on Thursday, September 13th in 283 Dwinelle at 7pm.
* Come learn more about the OCF
* Learn how to become a staff member
* Help elect the OCF General Manager and Site Manager
* Find out about our upcoming events

Also, stay afterwards for some food and socializing. We hope to see all of you there! diff --git a/docs/news/posts/blogger-post-2007-09-22-1.md b/docs/news/posts/blogger-post-2007-09-22-1.md new file mode 100644 index 0000000..df80495 --- /dev/null +++ b/docs/news/posts/blogger-post-2007-09-22-1.md @@ -0,0 +1,6 @@ +--- +date: 2007-09-22 +--- + +# Maintenance Day and Minor Downtime +The OCF will be closed on Sunday, September 23 for cleaning. Some lab workstations may be temporarily turned off throughout the day. Furthermore, there will be a slight disruption in our services between 2 and 3 PM while we install hardware into one of our core servers.

UPDATE: As some people have inquired about which services will be affected by the downtime, we anticipate that most of our mail services (IMAP/POP3/SMTP and mailing lists) will be affected by the downtime. diff --git a/docs/news/posts/blogger-post-2007-10-08-1.md b/docs/news/posts/blogger-post-2007-10-08-1.md new file mode 100644 index 0000000..e5ef15c --- /dev/null +++ b/docs/news/posts/blogger-post-2007-10-08-1.md @@ -0,0 +1,6 @@ +--- +date: 2007-10-08 +--- + +# General Meeting 3.0 +The OCF will be holding another general meeting this Thursday, October 11th, in 283 Dwinelle at 7 PM.

* Come learn more about the OCF
* Learn how to become a staff member
* Help elect the OCF Site Manager
* Find out about our exciting upcoming events

Also, stay afterwards for some food and socializing.

I hope to see all of you there! diff --git a/docs/news/posts/blogger-post-2007-10-08-2.md b/docs/news/posts/blogger-post-2007-10-08-2.md new file mode 100644 index 0000000..00c4cf0 --- /dev/null +++ b/docs/news/posts/blogger-post-2007-10-08-2.md @@ -0,0 +1,6 @@ +--- +date: 2007-10-08 +--- + +# Maintenance Day 2.0 +Due to scheduling issues with various staffers, the OCF lab maintenance that was scheduled to occur on Sunday, September 23, was postponed to Saturday, October 6. The lab cleaning was completed with great success, and OCF lab looks (almost) new!

I'd like to thank our General Manager, Angel Kittiyachavalit, for organizing the event and all the staffers who showed up to lend a helping hand. You guys are awesome! diff --git a/docs/news/posts/blogger-post-2007-10-18-1.md b/docs/news/posts/blogger-post-2007-10-18-1.md new file mode 100644 index 0000000..180ca37 --- /dev/null +++ b/docs/news/posts/blogger-post-2007-10-18-1.md @@ -0,0 +1,6 @@ +--- +date: 2007-10-18 +--- + +# Home Directories Deleted and Restored +During maintenance of our file server at 4 AM, the home directories of users whose usernames began with the letter 'a', letter 'b', or letters 'ca' through 'cg' were inadvertently deleted. The directories were promptly restored from a backup made an hour earlier. We apologize for the inconvenience.

I'll post up more details when I get a chance. diff --git a/docs/news/posts/blogger-post-2007-10-18-2.md b/docs/news/posts/blogger-post-2007-10-18-2.md new file mode 100644 index 0000000..5cc1551 --- /dev/null +++ b/docs/news/posts/blogger-post-2007-10-18-2.md @@ -0,0 +1,6 @@ +--- +date: 2007-10-18 +--- + +# Downtime 10/20 +There is scheduled downtime for 10/20 as we upgrade one of our core servers. Due to the recent error made in our test, we will be discussing whether or not to postpone the upgrade until our tests are clean. The decision will be made during BoD tonight.

Edit: We are going through with the upgrade with modified instructions so as to prevent what happened during the dry-run.

Edit: Timeline is available at http://xrl.us/7d88. diff --git a/docs/news/posts/blogger-post-2007-11-03-1.md b/docs/news/posts/blogger-post-2007-11-03-1.md new file mode 100644 index 0000000..68c8206 --- /dev/null +++ b/docs/news/posts/blogger-post-2007-11-03-1.md @@ -0,0 +1,6 @@ +--- +date: 2007-11-03 +--- + +# Harder Better Faster Stronger Webmail and Documentation +We just installed a caching module that'll greatly speed up our webmail and documentation pages. In some cases, you'll notice a speedup of about 2-3x!

While you're giving our enhanced documentation a test drive, take a look at our recently updated web hosting documentation. We've added a bunch of information, updated a lot of the existing documentation, and tried to clarify any confusing information.

We're also building up a list of suggested web applications that have been reported to work with our web server, along with any special customizations that might help them run better on the OCF.

This documentation refresh is an ongoing process, so keep checking back!

Oh, last but not least, we've added an auto-save feature to our webmail. Now, whenever you're writing an email, our webmail interface will automatically save the message every few seconds. No longer will you lose an email to a web browser crash or errant click! diff --git a/docs/news/posts/blogger-post-2008-01-31-1.md b/docs/news/posts/blogger-post-2008-01-31-1.md new file mode 100644 index 0000000..1c51164 --- /dev/null +++ b/docs/news/posts/blogger-post-2008-01-31-1.md @@ -0,0 +1,6 @@ +--- +date: 2008-01-31 +--- + +# First General Meeting for Spring 2008 +Welcome back OCFers!

We hope everyone is excited for a new semester with OCF.

Our first general meeting will be held on Thursday, February 7th in 109 Wheeler at 7pm.

Also, stay afterwards for some food and socializing. We hope to see all of you there! diff --git a/docs/news/posts/blogger-post-2008-02-29-1.md b/docs/news/posts/blogger-post-2008-02-29-1.md new file mode 100644 index 0000000..03c0422 --- /dev/null +++ b/docs/news/posts/blogger-post-2008-02-29-1.md @@ -0,0 +1,6 @@ +--- +date: 2008-02-29 +--- + +# The Long Promised Website Uploader +The OCF website manager, our SFTP web-based interface, has now entered beta mode. What it does is allow you to upload files to your webspace and your home directory through a purely web-based interface, in our efforts to make this process as easy as possible. I would like to thank Stephen Le for setting up this installation into something functional. I've also re-skinned the default "Blue" theme with a custom OCF theme that is more or less complete. In time, this will become a standard tool for all OCF users, once we make sure everything works smoothly. diff --git a/docs/news/posts/blogger-post-2008-03-12-1.md b/docs/news/posts/blogger-post-2008-03-12-1.md new file mode 100644 index 0000000..e72b52b --- /dev/null +++ b/docs/news/posts/blogger-post-2008-03-12-1.md @@ -0,0 +1,6 @@ +--- +date: 2008-03-12 +--- + +# PHP 4 to PHP 5 transition +Due to the PHP developers' decision to end support for PHP 4, the OCF will be requiring all users with PHP web applications, such as WordPress, phpBB, or Gallery, to use PHP 5. This transition, required by the campus Minimum Security Standards, will provide more features, security, and performance for PHP web applications on the OCF. As the current default PHP interpreter is PHP 4, almost all users of PHP will be affected, though few will need to make drastic changes to their applications. The transition will happen in two stages:

Users whose websites do not include web applications, or whose web applications are written in another language such as Perl, will not be affected.

All PHP users should review their web applications for compatibility with PHP 5 as soon as possible. Most widely-used web applications already support PHP 5, so in many cases, no action is needed; in most others, an application upgrade will be sufficient to ensure continued functionality. Users who use custom-written PHP applications need to review their code for PHP 5 compatibility; see the links provided in our documentation for details on what's changed.

We recommend that you test your web application with PHP 5 before the transition date; instructions on how to do this are in our documentation.

The latest information on the change, and details on how you can ensure your web applications continue to function, can be found at:

http://docs.ocf.berkeley.edu/wiki/Major_service_changes

If you have any questions or concerns, please don't hesitate to contact us (email staff@ocf.berkeley.edu). diff --git a/docs/news/posts/blogger-post-2008-03-15-1.md b/docs/news/posts/blogger-post-2008-03-15-1.md new file mode 100644 index 0000000..92ae2f1 --- /dev/null +++ b/docs/news/posts/blogger-post-2008-03-15-1.md @@ -0,0 +1,6 @@ +--- +date: 2008-03-15 +--- + +# Print Queue Online and In Real Life +We've talked about it, and now it's here! The print queue is now available on display in lab on a dedicated machine and monitor by the printer, as well as online at your convenience. It was formerly accessible only via command line (lpq -s) and printing status windows in Windows, neither of which were visible enough for the majority of our users to notice. Thank you Aaron Oaks, Genevieve Wong, and Mike Gasidlo for working on making this happen. diff --git a/docs/news/posts/blogger-post-2008-03-18-1.md b/docs/news/posts/blogger-post-2008-03-18-1.md new file mode 100644 index 0000000..fc48f05 --- /dev/null +++ b/docs/news/posts/blogger-post-2008-03-18-1.md @@ -0,0 +1,6 @@ +--- +date: 2008-03-18 +--- + +# OCF general meeting Thursday 3/20 +There will be a general meeting Thursday, March 20, at 7 PM in 109 Wheeler, to discuss amendments to the OCF bylaws and constitution. See http://www.ocf.berkeley.edu/~sluo/draft-bylaws.txt for details of the proposed changes. Everyone is welcome to attend, and food will be provided. diff --git a/docs/news/posts/blogger-post-2008-03-21-1.md b/docs/news/posts/blogger-post-2008-03-21-1.md new file mode 100644 index 0000000..f1ef2f8 --- /dev/null +++ b/docs/news/posts/blogger-post-2008-03-21-1.md @@ -0,0 +1,6 @@ +--- +date: 2008-03-21 +--- + +# Reminder: PHP 5 now the default +As previously announced, the default PHP interpreter is now PHP 5 as of today. See the previous post or our documentation for more information on what this means and how to ensure your web applications continue to work. diff --git a/docs/news/posts/blogger-post-2008-03-25-1.md b/docs/news/posts/blogger-post-2008-03-25-1.md new file mode 100644 index 0000000..af3e74f --- /dev/null +++ b/docs/news/posts/blogger-post-2008-03-25-1.md @@ -0,0 +1,6 @@ +--- +date: 2008-03-25 +--- + +# OCF downtime Thursday 3/27 from 10 AM to 5 PM +Eshleman Hall is scheduled to be dark this Thursday (3/27) from 10 AM to 5 PM for construction work. Since our connection to the campus network and the rest of the world runs through Eshleman, we'll be offline during this period. Incoming mail should be queued by senders and delivered normally after the outage, so it won't be lost. diff --git a/docs/news/posts/blogger-post-2008-04-18-1.md b/docs/news/posts/blogger-post-2008-04-18-1.md new file mode 100644 index 0000000..3aad17a --- /dev/null +++ b/docs/news/posts/blogger-post-2008-04-18-1.md @@ -0,0 +1,6 @@ +--- +date: 2008-04-18 +--- + +# LDAP and Kerberos Migration +As of Wednesday, April 16, the OCF has begun using a new and improved account storage and authentication framework. Among other things, this transition brings faster and more reliable Windows printing and single-sign-on support for the Unix clients. It also means that you MUST sign on to at least one OCF service (Windows or Unix clients, webmail, POP, IMAP, online account tools, SSH (WITH YOUR PASSWORD), etc.) sometime in the next six months, or else you will become an obligatory beta tester for the password migration tool, coming in the fall. diff --git a/docs/news/posts/blogger-post-2008-05-17-1.md b/docs/news/posts/blogger-post-2008-05-17-1.md new file mode 100644 index 0000000..34b79f1 --- /dev/null +++ b/docs/news/posts/blogger-post-2008-05-17-1.md @@ -0,0 +1,6 @@ +--- +date: 2008-05-17 +--- + +# Slow Web Services +Our web server is currently being overwhelmed by a flood of spam. We're working on blocking the spam and restoring our web services.

UPDATE: I think we've got everything under control. Our web services should be working normally now. Send us an email if you notice anything strange. diff --git a/docs/news/posts/blogger-post-2008-05-20-1.md b/docs/news/posts/blogger-post-2008-05-20-1.md new file mode 100644 index 0000000..be66eb5 --- /dev/null +++ b/docs/news/posts/blogger-post-2008-05-20-1.md @@ -0,0 +1,6 @@ +--- +date: 2008-05-20 +--- + +# A Sight for Sore Eyes +We've just rolled out a bunch of new widescreen monitors for our new workstations:

Dell 2009W

Besides an awesome 20" of viewable goodness and 1680x1050 resolution, you can also rotate the screen:

Dell 2009W

If you haven't visited our lab in awhile, come by and check out our new hardware, and if you don't already have an account, go get one! diff --git a/docs/news/posts/blogger-post-2008-05-23-1.md b/docs/news/posts/blogger-post-2008-05-23-1.md new file mode 100644 index 0000000..84ae1f3 --- /dev/null +++ b/docs/news/posts/blogger-post-2008-05-23-1.md @@ -0,0 +1,6 @@ +--- +date: 2008-05-23 +--- + +# Unplanned Network Outage +Our upstream network provider is having some network problems, rendering the OCF inaccessible. We're investigating the problem and trying to restore network access to the OCF as quickly as possible.

Thanks for your patience! diff --git a/docs/news/posts/blogger-post-2008-05-23-2.md b/docs/news/posts/blogger-post-2008-05-23-2.md new file mode 100644 index 0000000..649c99c --- /dev/null +++ b/docs/news/posts/blogger-post-2008-05-23-2.md @@ -0,0 +1,6 @@ +--- +date: 2008-05-23 +--- + +# PHP 4 Support Terminated +As announced in March, we have ended our support for PHP 4. Any remaining PHP 4 users must migrate their code to PHP 5. diff --git a/docs/news/posts/blogger-post-2008-05-24-1.md b/docs/news/posts/blogger-post-2008-05-24-1.md new file mode 100644 index 0000000..c15a112 --- /dev/null +++ b/docs/news/posts/blogger-post-2008-05-24-1.md @@ -0,0 +1,6 @@ +--- +date: 2008-05-24 +--- + +# Network Restored +The OCF's network access was just restored, and we're checking to see if everything works. I'll provide updates to this post if anything comes up. diff --git a/docs/news/posts/blogger-post-2008-05-26-1.md b/docs/news/posts/blogger-post-2008-05-26-1.md new file mode 100644 index 0000000..2a4b6f1 --- /dev/null +++ b/docs/news/posts/blogger-post-2008-05-26-1.md @@ -0,0 +1,6 @@ +--- +date: 2008-05-26 +--- + +# Harder Better Faster Stronger Webmail and Documentation (Redux) +We just upgraded our webmail and documentation server to a machine with more than four times the processing power and RAM. Combined with the changes we made last semester, our webmail and documentation should be faster than ever.

We haven't forgotten about our other services, though, and we're working on moving them to faster and more reliable hardware this summer. We know that many of you have been suffering with the occasional slowness in our services, and we're working on fixing it! Please be patient as upgrades to our other services require lots of planning and testing.

Before I go, I'd like to extend a special thank you to Sun Microsystems, who provided us with the hardware that made this upgrade possible. They make awesome server hardware! diff --git a/docs/news/posts/blogger-post-2008-06-09-1.md b/docs/news/posts/blogger-post-2008-06-09-1.md new file mode 100644 index 0000000..c693bce --- /dev/null +++ b/docs/news/posts/blogger-post-2008-06-09-1.md @@ -0,0 +1,6 @@ +--- +date: 2008-06-09 +--- + +# OCF Laboratory Move +The OCF laboratory in Heller Lounge will be closed throughout the day as we relocate to our new location in Eshleman Hall.

We apologize for the short notice. Please contact us via email or IRC if you have any questions or concerns. diff --git a/docs/news/posts/blogger-post-2008-09-01-1.md b/docs/news/posts/blogger-post-2008-09-01-1.md new file mode 100644 index 0000000..59f0e26 --- /dev/null +++ b/docs/news/posts/blogger-post-2008-09-01-1.md @@ -0,0 +1,6 @@ +--- +date: 2008-09-01 +--- + +# Unplanned Mail Downtime +At 3AM on September 1, we began having issues with the hardware hosting our mail. We are currently working on identifying the problem and minimizing its impact upon the rest of our services.

We will update this post once we have more information.

UPDATE: It looks like our disk array had some hard drive issues. We were using mirrored hard drives, so we don't think any data was affected. Everything should be working again, but please let us know if you see anything out of the ordinary. diff --git a/docs/news/posts/blogger-post-2008-09-11-1.md b/docs/news/posts/blogger-post-2008-09-11-1.md new file mode 100644 index 0000000..9867526 --- /dev/null +++ b/docs/news/posts/blogger-post-2008-09-11-1.md @@ -0,0 +1,6 @@ +--- +date: 2008-09-11 +--- + +# FREE FOOD! +The OCF is holding a General Meeting on Thursday, September 11 at 7PM in 263 Dwinelle.

There will be free food! Come hear the latest news about our lab relocation and meet the staff who keep your services running! diff --git a/docs/news/posts/blogger-post-2008-09-19-1.md b/docs/news/posts/blogger-post-2008-09-19-1.md new file mode 100644 index 0000000..f8287ed --- /dev/null +++ b/docs/news/posts/blogger-post-2008-09-19-1.md @@ -0,0 +1,6 @@ +--- +date: 2008-09-19 +--- + +# Buzz +Check us out at The Daily Clog for an outsider perspective on our Heller-to-Eshleman relocation situation. Free computing is becoming a little hard to come by these days, but we will be back and better before the first snow falls. (Luckily, we're in California.) diff --git a/docs/news/posts/blogger-post-2008-09-28-1.md b/docs/news/posts/blogger-post-2008-09-28-1.md new file mode 100644 index 0000000..5beacb0 --- /dev/null +++ b/docs/news/posts/blogger-post-2008-09-28-1.md @@ -0,0 +1,6 @@ +--- +date: 2008-09-28 +--- + +# OCF System Maintenance Fall 2008 +The OCF is down for maintenance. We don't anticipate that it will last more than 12 hours, but if it does, we will post updates here. diff --git a/docs/news/posts/blogger-post-2008-10-07-1.md b/docs/news/posts/blogger-post-2008-10-07-1.md new file mode 100644 index 0000000..a88df89 --- /dev/null +++ b/docs/news/posts/blogger-post-2008-10-07-1.md @@ -0,0 +1,6 @@ +--- +date: 2008-10-07 +--- + +# iPhone-tailored OCF website +Been thinking about doing this for a while now, and as of today, owners of the iPhone and iPod Touch can point their devices to: ocf.berkeley.edu/iphone for a slimmed down essentials-only mobile version designed specifically for this viewport. There are some goodies in there, including the phone number linking to your dial pad, or the map location linking to your built-in Google Maps.

I've tested it against the iPhone SDK's simulator, and against Safari in general, and we'd love any feedback you have. For those of you without these devices, you can play around with it with TestiPhone.com, though it should work with most devices with similar viewports. Or you can simply view screenshots. And of course, you always have the option to view the full versions of these pages in these devices.

I have plans to develop a more general mobile version for other devices as time permits. Keep posted. Coming to a device near you. diff --git a/docs/news/posts/blogger-post-2008-10-14-1.md b/docs/news/posts/blogger-post-2008-10-14-1.md new file mode 100644 index 0000000..7628740 --- /dev/null +++ b/docs/news/posts/blogger-post-2008-10-14-1.md @@ -0,0 +1,6 @@ +--- +date: 2008-10-14 +--- + +# Unplanned Downtime +We began having problems communicating with our disk array this morning. Since most of our data is stored on the disk array, many of our services are currently offline.

We are working on fixing the disk array. The data on the disk array is backed up daily, so even if the disk array is broken, all data should be safe.

UPDATE: We've brought back all our services as of 3:30 PM. Let us know if anything is behaving strangely. diff --git a/docs/news/posts/blogger-post-2008-10-18-1.md b/docs/news/posts/blogger-post-2008-10-18-1.md new file mode 100644 index 0000000..3a20abf --- /dev/null +++ b/docs/news/posts/blogger-post-2008-10-18-1.md @@ -0,0 +1,6 @@ +--- +date: 2008-10-18 +--- + +# Planned Maintenance +The OCF is currently down for maintenance. Some of you may have noticed the unplanned downtime this past Tuesday, which was related to errors with our disk array, which houses mail as well as many of the critical files for the OCF. This downtime is meant to help fix some of those problems and stabilize our servers.

We don't expect it to last past 9 AM on Sunday, October 19. diff --git a/docs/news/posts/blogger-post-2008-10-19-1.md b/docs/news/posts/blogger-post-2008-10-19-1.md new file mode 100644 index 0000000..9103286 --- /dev/null +++ b/docs/news/posts/blogger-post-2008-10-19-1.md @@ -0,0 +1,6 @@ +--- +date: 2008-10-19 +--- + +# Maintenance Complete +We've finished our maintenance ahead of schedule, and all OCF services should be working again.

Thanks for your patience and understanding! diff --git a/docs/news/posts/blogger-post-2008-10-23-1.md b/docs/news/posts/blogger-post-2008-10-23-1.md new file mode 100644 index 0000000..ad77e31 --- /dev/null +++ b/docs/news/posts/blogger-post-2008-10-23-1.md @@ -0,0 +1,6 @@ +--- +date: 2008-10-23 +--- + +# Power Outage +There will be a power outage in the building housing our core servers between 1-6 AM on Thursday, October 23. We anticipate shutting down our servers around midnight tonight and powering them back on as soon as we can.

I apologize for the short notice, but we were just notified today. diff --git a/docs/news/posts/blogger-post-2008-11-01-1.md b/docs/news/posts/blogger-post-2008-11-01-1.md new file mode 100644 index 0000000..b723d37 --- /dev/null +++ b/docs/news/posts/blogger-post-2008-11-01-1.md @@ -0,0 +1,6 @@ +--- +date: 2008-11-01 +--- + +# Login Server Upgrades +We just upgraded the operating system of one of our login servers, conquest.OCF.Berkeley.EDU. The machine now is available for general use and will be re-added to our login server alias once we're sure that everything works. Feel free to give the server a test drive, but please note that we might have to reboot it a few times in the coming days.

This operating system upgrade allows us to deliver greater security and better application support to our users. In fact, I'm pleased to announce that we now support Mercurial, a popular distributed version control system.

We'll also be upgrading apocalypse.OCF.Berkeley.EDU soon. If you really need a place to work uninterrupted this weekend, you can try our Linux login server, tsunami.OCF.Berkeley.EDU. diff --git a/docs/news/posts/blogger-post-2009-01-27-1.md b/docs/news/posts/blogger-post-2009-01-27-1.md new file mode 100644 index 0000000..04081c0 --- /dev/null +++ b/docs/news/posts/blogger-post-2009-01-27-1.md @@ -0,0 +1,6 @@ +--- +date: 2009-01-27 +--- + +# General Meeting on February 5 +The OCF will be holding a general meeting next Thursday, February 5th, at 7 PM in 189 Dwinelle. Hear the latest updates on our lab relocation, meet the staff who volunteer daily to keep your services running, and elect this semester's OCF managers.

Everyone's invited, so come on by for food and socializing!
diff --git a/docs/news/posts/blogger-post-2009-02-04-1.md b/docs/news/posts/blogger-post-2009-02-04-1.md new file mode 100644 index 0000000..49ab28f --- /dev/null +++ b/docs/news/posts/blogger-post-2009-02-04-1.md @@ -0,0 +1,6 @@ +--- +date: 2009-02-04 +--- + +# Unexpected Downtime +We're currently experiencing some hardware problems on one of our core servers. We are investigating the issue and will post updates here as we determine the scope of the problem.

UPDATE: Disk array issues again. We're running a check on the data, and we expect to be back online tomorrow morning. Sorry for the inconvenience. We understand that it's a school night, so please let us know (sm @ ocf.berkeley.edu) if you need emergency access to your files.

UPDATE (1:01 AM): The check completed successfully, and we're bringing all OCF services back right now. Let us know if you notice anything strange with your files.
diff --git a/docs/news/posts/blogger-post-2009-02-06-1.md b/docs/news/posts/blogger-post-2009-02-06-1.md new file mode 100644 index 0000000..688c383 --- /dev/null +++ b/docs/news/posts/blogger-post-2009-02-06-1.md @@ -0,0 +1,6 @@ +--- +date: 2009-02-06 +--- + +# OCF Lab Reopening! +The OCF lab will be reopening on Monday, February 9. We have a shiny new lab in the basement of Eshleman Hall, a cool lounge just for hanging out, and Airbears!

So come by to check it out next week, and thank you for being so patient with us for the past seven months!
diff --git a/docs/news/posts/blogger-post-2009-02-25-1.md b/docs/news/posts/blogger-post-2009-02-25-1.md new file mode 100644 index 0000000..64df9be --- /dev/null +++ b/docs/news/posts/blogger-post-2009-02-25-1.md @@ -0,0 +1,6 @@ +--- +date: 2009-02-25 +--- + +# Yes We Can Upgrade Quotas +
Good news: we're providing storage stimulus by upgrading all user disk space quotas to 2 GB.

Bad news: to enact this change, the OCF will be unavailable between 12 AM, Friday, February 27 and 8 AM, Monday, March 2 (just over 72 hours).

We know three days is a long time, but the prolonged downtime is necessary to ensure the safety of user data while we perform our upgrades. Delivering data reliability you can believe in is important to us.

We're hoping this will be the first of many upgrades for our users. Who knows? With the new lab, this could signal a new beginning. 
diff --git a/docs/news/posts/blogger-post-2009-03-11-1.md b/docs/news/posts/blogger-post-2009-03-11-1.md new file mode 100644 index 0000000..485d230 --- /dev/null +++ b/docs/news/posts/blogger-post-2009-03-11-1.md @@ -0,0 +1,6 @@ +--- +date: 2009-03-11 +--- + +# RIP Logjam, You Were a Good Printer +logjam, our HP LaserJet 4350 laser printer, died over the weekend. It's out of warranty, so we're currently exploring our options. In the mean time, we've brought one of our old printers back into service.

We apologize for the disruption in service and slower printing you may experience while we work on fixing/replacing our primary printer.
diff --git a/docs/news/posts/blogger-post-2009-03-30-1.md b/docs/news/posts/blogger-post-2009-03-30-1.md new file mode 100644 index 0000000..bb6e826 --- /dev/null +++ b/docs/news/posts/blogger-post-2009-03-30-1.md @@ -0,0 +1,6 @@ +--- +date: 2009-03-30 +--- + +# Unexpected downtime +Sometime this morning, our core fileserver started having some disk issues. We're working on figuring out what exactly went wrong and fixing it, but in the meantime, the OCF is effectively down (no lab, no SSH, no web, etc). We'll post updates as we have them.

UPDATE: The problem seems to have been fixed, though we're still not entirely sure what it was. We'll keep working on it throughout the day. In the meantime, though, the OCF is back up and ready for business. diff --git a/docs/news/posts/blogger-post-2009-04-07-1.md b/docs/news/posts/blogger-post-2009-04-07-1.md new file mode 100644 index 0000000..7d02be1 --- /dev/null +++ b/docs/news/posts/blogger-post-2009-04-07-1.md @@ -0,0 +1,6 @@ +--- +date: 2009-04-07 +--- + +# New Printers +We've purchased and installed two Xerox Phaser 4510DT laser printers to replace our old, broken, printer.

Let us know if you have any problems using the new printers! diff --git a/docs/news/posts/blogger-post-2009-04-25-1.md b/docs/news/posts/blogger-post-2009-04-25-1.md new file mode 100644 index 0000000..1790d40 --- /dev/null +++ b/docs/news/posts/blogger-post-2009-04-25-1.md @@ -0,0 +1,6 @@ +--- +date: 2009-04-25 +--- + +# OCF downtime +The OCF login servers (conquest and apocalypse) and lab machines are currently down for maintenance, and many services in the lab itself are unavailable. We apologize for the inconvenience and the delay in communicating this, and hope to be able to provide more details soon. diff --git a/docs/news/posts/blogger-post-2009-04-29-1.md b/docs/news/posts/blogger-post-2009-04-29-1.md new file mode 100644 index 0000000..59afed6 --- /dev/null +++ b/docs/news/posts/blogger-post-2009-04-29-1.md @@ -0,0 +1,6 @@ +--- +date: 2009-04-29 +--- + +# None +We apologize for the lack of notification and the lack of status updates, and are grateful for your patience. As you may have guessed, this is not a planned outage.

On April 19th, we discovered that one of our servers had been broken into. We initially tried to contain the problem, but further investigation revealed that the scope of the break-in was worse than we had originally thought. At our Board of Directors meeting on Thursday, April 23, it was decided that many, if not all, of the OCF servers would need to be rebuilt, to reduce the possibility that the hacker would be able to break back into our systems. Because of the nature of the incident, in the interests of security and damage control, the decision was made to begin reconstruction efforts immediately, rather than waiting the customary week to give notification.

Starting that night, various servers have been pulled from the network for forensic analysis and re-installation. However, we have been delayed in restarting some of our services due to an unforeseen difficulty in rebuilding one of our core servers. Until that issue is cleared up, we will be unable to resume anything approaching normal service.

For those of you worried about mail, the mail servers are continuing to accept incoming mail, and it will be available for pick-up once normal services resume.

I'm very sorry that I cannot give you a more solid timeframe than "soon." Please, rest assured that we are doing our very best to bring the OCF back online, and all of our senior technical staff are hard at work fixing this roadblock to normalcy. We ask your patience and understanding, noting that all our staff are volunteers and students who, like you, have class and work obligations, especially with finals looming ever closer.

As always, you are welcome to join us on the OCF IRC channel, irc.ocf.berkeley.edu, with questions, concerns, or anything else you think we should know. diff --git a/docs/news/posts/blogger-post-2009-05-06-1.md b/docs/news/posts/blogger-post-2009-05-06-1.md new file mode 100644 index 0000000..906311d --- /dev/null +++ b/docs/news/posts/blogger-post-2009-05-06-1.md @@ -0,0 +1,6 @@ +--- +date: 2009-05-06 +--- + +# OCF downtime: frequently asked questions +We've been getting a lot of questions over the past week, and (understandably) many of them are similar, so I thought I'd collect the answers to the most frequently asked ones here.

We thank you for your patience, and we'll continue to provide updates via our main website and this blog as things change and we get back on our feet.

Q: Can I access my email?
A: As of a few hours ago, there's a temporary webmail interface available at https://webmail.ocf.berkeley.edu/. Sending email from it doesn't work for the moment, and it lacks the spit and polish of the usual webmail interfaces, but you should at least be able to read your email. POP and IMAP access is still available, and you can send email using SMTP -- see our documentation (now also available again) for information on how to set this up.

Q: Can I access my files on the OCF?
Q: Can I modify my website hosted on the OCF?
A: The usual SSH and SFTP access is unavailable at the moment. For now, you can use the web FTP application to upload, download, and edit your files.

Q: What happened to my account application?
A: We can't process account applications at the moment. We'll get around to it once we have our services up and running.

Q: Can I use SSH or SFTP?
Q: I tried to log in via SSH/SFTP, but I'm getting a message that the host key changed.
A:The primary login servers that you get by logging into ocf.berkeley.edu are turned on again, but aren't accepting user logins at the moment. We'll let you know when you can log in to these machines again. The host keys have changed, and we'll post the new key fingerprints, along with instructions, when we're ready to allow logins again.

Q: So what's this about being hacked? Should I do anything to make sure I'm safe?
A: We recommend the following:

We will be requiring that all users change their OCF account passwords soon, but you cannot do this now, nor do we recommend doing it now.

If you have other questions, feel free to email us (staff [AT] ocf.berkeley.edu). diff --git a/docs/news/posts/blogger-post-2009-05-07-1.md b/docs/news/posts/blogger-post-2009-05-07-1.md new file mode 100644 index 0000000..c70417f --- /dev/null +++ b/docs/news/posts/blogger-post-2009-05-07-1.md @@ -0,0 +1,6 @@ +--- +date: 2009-05-07 +--- + +# apocalypse accepting user logins +apocalypse.OCF.Berkeley.EDU (192.58.221.245) is now accepting user logins. You should be able to connect to the server via SSH/SFTP to access your home directory and web space. If you're comfortable using a command-line mail client (e.g. mutt or pine), you'll also be able to read and send mail.

There are a few lingering issues with the server. In particular:
The SSH host keys are:
Thanks for your continued patience! diff --git a/docs/news/posts/blogger-post-2009-06-12-1.md b/docs/news/posts/blogger-post-2009-06-12-1.md new file mode 100644 index 0000000..f0a6daf --- /dev/null +++ b/docs/news/posts/blogger-post-2009-06-12-1.md @@ -0,0 +1,6 @@ +--- +date: 2009-06-12 +--- + +# Unexpected Downtime +OCF servers and services are currently experiencing unexpected downtime, due to what may be a Denial of Service (DoS) attack on the subnet the OCF is on. Most staffers are currently out of town due to the summer break; while this is being looked into, it's a bit difficult without connectivity to the servers.

Further updates to come on this blog. While our Internet Relay Chat (IRC) server is down, you can connect to CSUA's IRC server at irc.csua.berkeley.edu and join us in #ocf for updates (mibbit IRC client).

2:18 PM: It seems like we're back online, but the causes of the outage is still unknown. No data loss is anticipated; please let us if otherwise. diff --git a/docs/news/posts/blogger-post-2009-06-18-1.md b/docs/news/posts/blogger-post-2009-06-18-1.md new file mode 100644 index 0000000..fb8b066 --- /dev/null +++ b/docs/news/posts/blogger-post-2009-06-18-1.md @@ -0,0 +1,6 @@ +--- +date: 2009-06-18 +--- + +# The Pre joins OCF Mobile +Last year, we launched our mobile site to load for the iPhone/iPod touch, and right after included other devices like the Blackberry Storm, Android-based G1, and Opera Mini (such as on the Blackberry Curve beyond the BB browser). All other devices could still enter the site manually through m.ocf.berkeley.edu.

The Palm Pre has now been added to that list. Data detection works just as it does with the iPhone OS, so your phone dialer, Google Maps, and email client will work with our pages there too. I'll be on the lookout for Windows Mobile devices to test upon.

diff --git a/docs/news/posts/blogger-post-2009-07-24-1.md b/docs/news/posts/blogger-post-2009-07-24-1.md new file mode 100644 index 0000000..2312d06 --- /dev/null +++ b/docs/news/posts/blogger-post-2009-07-24-1.md @@ -0,0 +1,6 @@ +--- +date: 2009-07-24 +--- + +# OCF Downtime Starting 7/28 +The OCF will be down next Tuesday due to electrical work being done in Eshleman Hall. Unfortunately, we have not been given a time frame, so servers may be down for up to a week. We apologize for the inconvenience, but will try hard to make sure services come back up as soon and as smoothly as possible. diff --git a/docs/news/posts/blogger-post-2009-08-06-1.md b/docs/news/posts/blogger-post-2009-08-06-1.md new file mode 100644 index 0000000..31b1cd3 --- /dev/null +++ b/docs/news/posts/blogger-post-2009-08-06-1.md @@ -0,0 +1,6 @@ +--- +date: 2009-08-06 +--- + +# Continuing Downtime +The downtime that began on 7/28 is expected to continue for a few more days as campus electricians procure parts to complete the electrical work. If there are no other unanticipated delays, we should be back up in the coming days.

We apologize for the downtime and inconvenience.

UPDATE: The electrical work will be completed by Friday, August 14, at 4 PM. Unfortunately, we have very limited staff presence in the Berkeley area due to the summer vacation, so it may take some time to get everything up and running.

SECOND UPDATE (14-Aug-2009, 3 PM): The electrical work is mostly complete, and power has been restored to our server room. Unfortunately, something happened to our network uplink during the downtime (maybe someone forgot to turn a switch back on), so we have no Internet access. We're looking into it.

THIRD UPDATE (18-Aug-2009, 4 PM): A campus network technician is working on restoring connectivity to the OCF. The cause of the problem is still undetermined, but we're pressuring them to get things fixed ASAP. diff --git a/docs/news/posts/blogger-post-2009-08-20-1.md b/docs/news/posts/blogger-post-2009-08-20-1.md new file mode 100644 index 0000000..5c2aee9 --- /dev/null +++ b/docs/news/posts/blogger-post-2009-08-20-1.md @@ -0,0 +1,6 @@ +--- +date: 2009-08-20 +--- + +# You Know What They Say About Academia... +The cause of our networking outage has not been determined. We're working with campus technicians to debug the problem, but everything is going really s-l-o-w-l-y. We're just as upset as you are about the prolonged downtime, but, as a student organization, we just don't have as much sway as a campus department. diff --git a/docs/news/posts/blogger-post-2009-08-22-1.md b/docs/news/posts/blogger-post-2009-08-22-1.md new file mode 100644 index 0000000..487bd0e --- /dev/null +++ b/docs/news/posts/blogger-post-2009-08-22-1.md @@ -0,0 +1,6 @@ +--- +date: 2009-08-22 +--- + +# Good News!! (About Time...) +Today, the electrical work in our new server room was finally finished! Also, our network problems are over! Connectivity has been restored - all thanks to IST, who swooped in today to hook OCF up to their own network. Not only do we receive a steady fiber connection, but we also get IPv6 and our very own subnet.

So why aren't OCF services up and running, you ask? Well, all the IP addresses have changed because we are on a different subnet. This means that in addition to flashing all the switches, we must also renumber all the IPs on our servers, which we have many of. This is quite a task, requiring staff members to sit in the server room with a screen and keyboard and reconfigure every single server we have. Since a couple staff members are returning to Berkeley this weekend, I am planning to do this on Monday so that they can help me out. And since everything that can go wrong already has, I am going to be optimistic and hope that we won't run into any problems bringing servers online. If all goes well, OCF can be back up by Tuesday morning! diff --git a/docs/news/posts/blogger-post-2009-08-27-1.md b/docs/news/posts/blogger-post-2009-08-27-1.md new file mode 100644 index 0000000..40f9be2 --- /dev/null +++ b/docs/news/posts/blogger-post-2009-08-27-1.md @@ -0,0 +1,6 @@ +--- +date: 2009-08-27 +--- + +# Status of OCF Services +We've been working hard to get all our services restored. As usual, we've run into some bumps along the way, but I'm happy to report that we've been able to bring up the following services:
Once again, we apologize for the extended downtime. We're exploring better ways of handling extended outages in the future, but we want to emphasize that this downtime was really an anomaly. A perfect storm of factors came together: a long power outage, a server room migration, a network disconnection, and an IP space change.

We'd also like to thank the wonderful people at Physical Plant, IST, Residential Computing, and other campus organizations for coming to our aid. Also, thanks to our users for their supportive and critical comments (hey, we deserve it!).

UPDATE (28-Aug-09, 9:30 AM): We had a bit of a hiccup in our services yesterday night. All services that were working yesterday morning should also be working now.

UPDATE (29-Aug-09, 7:00 PM): We're still restoring our mail services. We haven't forgotten about our other services (printing, web file access, etc.) We're just prioritizing our most important ones.

UPDATE (1-Sep-09, 9:00 AM): We're starting to ramp up incoming mail. Once we're sure that it's working perfectly, we'll begin restoring the rest of our services.

UPDATE (2-Sep-09, 8:00 AM): Incoming mail is completely operational. We're starting work on restoring the rest of our services. diff --git a/docs/news/posts/blogger-post-2009-09-06-1.md b/docs/news/posts/blogger-post-2009-09-06-1.md new file mode 100644 index 0000000..2722b60 --- /dev/null +++ b/docs/news/posts/blogger-post-2009-09-06-1.md @@ -0,0 +1,7 @@ +--- +date: 2009-09-06 +--- + +# General Meeting on 9/10 +Hello dear OCF members!
+The new school year has begun, so it is time again for the OCF’s semi-annual General Meeting! It will be held at 7PM in 243 Dwinelle Hall. Come vote for this semester's new General and Site Managers, give us your suggestions, volunteer to help (we obviously need it), or just pelt us with your complaints! Pizza and various drinks shall be served to enhance the experience. See you all there! diff --git a/docs/news/posts/blogger-post-2009-09-16-1.md b/docs/news/posts/blogger-post-2009-09-16-1.md new file mode 100644 index 0000000..cf3c6a8 --- /dev/null +++ b/docs/news/posts/blogger-post-2009-09-16-1.md @@ -0,0 +1,12 @@ +--- +date: 2009-09-16 +--- + +# Unexpected Downtime +The OCF directory server went down today (Tuesday 9/15) at roughly 10:20 PM, taking pretty much everything else with it. We're still working on figuring out what went wrong and how to fix it.
+
+We'll post updates here as we have them. You can also find us at on the #OCF channel on the CSUA irc server at irc.csua.berkeley.edu.
+
+UPDATE 9/16/09 3:00 PM: It looks like everything is back up again. If you find anything missing or broken, please let us know.
+
+And to everyone who was wondering, yes, this is why your passwords/email stopped working. This is also indirectly responsible for why you may have gotten funny looking prompts when logging in remotely. diff --git a/docs/news/posts/blogger-post-2009-11-17-1.md b/docs/news/posts/blogger-post-2009-11-17-1.md new file mode 100644 index 0000000..3074613 --- /dev/null +++ b/docs/news/posts/blogger-post-2009-11-17-1.md @@ -0,0 +1,6 @@ +--- +date: 2009-11-17 +--- + +# Emergency Webserver Maintenance +At around 8:30 PM this Monday, OCF staff noticed that the OCF website was slow to respond, and that our webserver either the victim of a DDOS attack or else swap space exhaustion. In any event, we've gone ahead and rebooted it. We will also use this opportunity to apply a few long-overdue patches to the machine.

We expect this downtime to last no longer than an hour. During this outage, most OCF services will be unavailable (the machine hosting our webserver also hosts a number of other critical services).

We apologize for the inconvenience and thank our users for their understanding.

UPDATE: We have since discovered a problem with the server hosting mail services. We are working on this issue now; we anticipate that this will add no more than an additional hour to our original schedule.

UPDATE 2: It looks like everything is back online as of 9:55 PM. Let us know if you discover anything out of the ordinary.

UPDATE 3: It looks like webmail is having some issues. We are investigating this issue, but have no timeframe for fixing it. UPDATE TO THE UPDATE: Looks like this was an easier fix than I had thought. Webmail should be working again.
diff --git a/docs/news/posts/blogger-post-2009-12-21-1.md b/docs/news/posts/blogger-post-2009-12-21-1.md new file mode 100644 index 0000000..be36da5 --- /dev/null +++ b/docs/news/posts/blogger-post-2009-12-21-1.md @@ -0,0 +1,6 @@ +--- +date: 2009-12-21 +--- + +# Unexpected Downtime +
Sometime this morning, one of the OCF's mail servers stopped responding to user logins. We are working as fast as we can to resolve the issue, but do not have a time estimate. The server that hosts webmail also recently stopped responding; we are working on restoring it, but are focusing on bringing mail services back online.

The mail server in question — mail.OCF.Berkeley.EDU — provides IMAP, POP, and SMTP access to mail, but stores users' email on the central file server. Old mail is still accessible via command-line mail clients like mutt and pine on our login servers (apocalypse and tsunami), and incoming mail will be queued on a different server and delivered when mail.OCF is restored.

We will post updates when we have them. If you need assistance, someone in the OCF IRC channel may be able to assist you, but please be patient if nobody responds right away. :) We apologize for the downtime and inconvenience, and wish you a wonderful winter break.

UPDATE (23 December): mail.OCF is partially online. You can access your mail with an IMAP or POP client such as Mozilla Thunderbird, but we are still ironing out some issues with sending mail — please let us know if you have any problems. Webmail is still down, but we are hoping to have that service restored within the next few days.

UPDATE (27 December): We've restored the server that hosts webmail and documentation. Thank you for your patience — have a happy New Year!

UPDATE (30 December): Our SSL certificate issuing authority, ipsCA, kinda dropped the ball and forgot to renew their root certificates before they expired on December 29th. As a result, unless you're using IE8, you will get a warning about an untrusted SSL certificate whenever you try to access OCF services like POP, IMAP, SMTP, Webmail, or account tools. We are working on obtaining updated certificates from a slightly more on-the-ball certificate authority, but in the meantime, you may continue accessing your OCF services by overriding the certificate warnings. However, in general this is a bad practice to get in the habit of, and we suggest that you wait until we have installed the updated certificates before attempting to connect.

UPDATE (5 January): We have installed the new SSL certs for mail.ocf and webmail.ocf. Let us know if you have any issues connecting to these services, and happy new year!
diff --git a/docs/news/posts/blogger-post-2010-02-13-1.md b/docs/news/posts/blogger-post-2010-02-13-1.md new file mode 100644 index 0000000..a19b9fe --- /dev/null +++ b/docs/news/posts/blogger-post-2010-02-13-1.md @@ -0,0 +1,6 @@ +--- +date: 2010-02-13 +--- + +# Patch Day +The OCF's patch day is set to begin in about half an hour, and is scheduled to run for 24 hours (6 PM Friday to 6 PM Saturday). Various services will drop in and out during that time, including mail, web, and SSH access. We'll post status updates here and will let you know about any unforseen events which would prolong the outage.

8 PM: The mail servers have been updated successfully, with no outage required thus far. We will need to shut down mail delivery, POP, and IMAP when we take the fileserver down later tonight, however. The login server and documentation server are taking longer than expected to patch; we'll hold of on the fileserver until those two are resolved.

10:45 PM: The final updates for the login and documentation servers have been applied, apparently with no adverse effects.  We've just taken mail down in preparation for patching our main fileserver.  We will be bringing down web next before commencing the patching.

1:45 AM: Looks like the bulk of the patching work is done.  There might be a few more short outages throughout the day as we tweak settings and ferret out bugs, but there should be no more prolonged outages (cross your fingers).
diff --git a/docs/news/posts/blogger-post-2010-04-05-1.md b/docs/news/posts/blogger-post-2010-04-05-1.md new file mode 100644 index 0000000..182cd35 --- /dev/null +++ b/docs/news/posts/blogger-post-2010-04-05-1.md @@ -0,0 +1,6 @@ +--- +date: 2010-04-05 +--- + +# Unexpected Downtime +The OCF's central file server went down today around 4:30, and we're having a bit of trouble nursing it back to a working state. We'll post updates here as we have them. diff --git a/docs/news/posts/blogger-post-2010-05-04-1.md b/docs/news/posts/blogger-post-2010-05-04-1.md new file mode 100644 index 0000000..d3ce7d3 --- /dev/null +++ b/docs/news/posts/blogger-post-2010-05-04-1.md @@ -0,0 +1,6 @@ +--- +date: 2010-05-04 +--- + +# Sorry... +Our Incoming mail server was having some difficulties with disk-space, somewhere in the stress of finals and the stress of having to resurrect the server, I panicked and accidentally deleted some queued up mail. So mail might behave a bit weird for the next day or two. Sorry for any inconvenience I may have caused you.

Sanjay Krishnan
OCF Site Manager diff --git a/docs/news/posts/blogger-post-2010-05-26-1.md b/docs/news/posts/blogger-post-2010-05-26-1.md new file mode 100644 index 0000000..6201c6e --- /dev/null +++ b/docs/news/posts/blogger-post-2010-05-26-1.md @@ -0,0 +1,6 @@ +--- +date: 2010-05-26 +--- + +# Summer Goings-On +So, some of our users have been concerned about the planned downtime this summer. Here's the nitty-gritty:

We're rebuilding everything from the ground up. Partly this is because the current system has things scattered over multiple servers, a relic of the days when we didn't have powerful enough systems to consolidate mail, for example, onto one machine. Partly this is because Oracle (formerly Sun) can no longer be relied upon to provide free security updates for Solaris, which forms the majority of our back-end. Finally, this is partly because the current setup is simply untenable. It's the computer equivalent of an Land Rover held together with spit, baling wire, and chewing gum - it runs, more or less, but God help anyone who needs to poke around under the hood when things break.

So, we decided to move on. Rather than spending time and energy trying to keep fixing a 20 year old heap, we're starting fresh. We'll be changing some things on the backend - most notably a migration from Solaris to Debian and FreeBSD - and we'll be changing some things on the frontend - like bringing in Windows 7 and getting some new hardware for the Linux clients. For most of our users, the change will be mostly transparent. For some of our users, things will change a little bit. For a tiny minority, things will break. To those people, we apologize in advance, and we are, as always, happy to help you get things working again.

In theory, while we're making this transition, we'll be building replacements side-by-side with the current production servers, and swapping them out once we're fairly certain that everything's working correctly. So, for the majority of our services, there won't be much more than a blip in service. Moreover, we won't be swapping out more than one server at a time, so no more than one service should go offline at any given time. However, there are some services (notably web and MySQL) which will take longer to swap out. The fact of the matter is that we really only have one server powerful enough to be a web server, so we can't build it's replacement until we've shut it down. Even so, our daring team of sysadmins should have the server back up and running in no time flat (I believe the previous record for a ground-up rebuild of the webserver was less than a day).

Finally, I'd like to bring some attention to the "in theory" that started off that last paragraph. As anyone who's ever worked on any sort of project before knows, something will always go wrong. So we ask you to bear with us while we work out the kinks and the bugs. This is going to take at least a few weeks, and things will be a little hectic during that time. We may swap in a new system only to find some bug that escaped our testing, and we'll have to switch back until we get it sorted out. We'll do our best to post here when we're getting ready to swap something out, and we'll make sure that there are always avenues open to get in touch with us to let us know about problems.

Oh- and remember that we're all volunteers. We do our best, but sometimes other commitments (school, work, family, life) can get in the way for a little while. But like all true geeks, we can't stay away for long, so rest assured that things will get fixed, emails will get answered, and the agents of truth and light will win the day.

Thanks for reading.
diff --git a/docs/news/posts/blogger-post-2010-05-27-1.md b/docs/news/posts/blogger-post-2010-05-27-1.md new file mode 100644 index 0000000..f716b8a --- /dev/null +++ b/docs/news/posts/blogger-post-2010-05-27-1.md @@ -0,0 +1,6 @@ +--- +date: 2010-05-27 +--- + +# First Update on Rebuild +Update 1 on OCF Rebuild

-Set up a new nat server and a new printhost
-Each one is a separate esxi guest
-Testing to see if I allocated enough memory, I think I may want to bump the printserver up to 512 MB of virtual ram (any thoughts?)
-New Lab IP ranges (formalized)
- So lab networking is being cleaned up, mostly wiring, I'm hoping to axe the windows vlan altogether since if you include the random shit we have acquired over the past year thats way more ports than what we traditionally allocated for the lab nat. (motivation for cleaning those IP's up)
-haven't set up any printers yet, one of them is possibly broken (or jammed)
-Hopefully next week we will start rolling out machine images...will start with linux
-8 pages of documentation so far, ask me if you want to read it. I'm doing them on word documents for now (not leaving any details out). traditionally we tend to leave stuff out like iptables and stuff out of staff wiki documentation, but staffers have messed stuff like that up before, so i'm not leaving even "trivial" stuff out...
-^perhaps we can make an ocf staff handbook or something basically command by command how to rebuild the ocf? diff --git a/docs/news/posts/blogger-post-2010-06-14-1.md b/docs/news/posts/blogger-post-2010-06-14-1.md new file mode 100644 index 0000000..09f0293 --- /dev/null +++ b/docs/news/posts/blogger-post-2010-06-14-1.md @@ -0,0 +1,6 @@ +--- +date: 2010-06-14 +--- + +# Unexpected Downtime +As many of you have noticed, the OCF is currently down. One of our core servers experienced a hardware failure earlier today and we are working to restore it. We do not yet have an ETA for restored service, but will post updates as we have them. We sincerely apologize for the inconvenience.

UPDATE (16 June, 1 PM): The failed server has been partly restored. Web and SSH logins are intermittently working for usernames starting with the letters a-p, but mail is still down systemwide; we hope to have full functionality soon. As always, you can find us in the OCF's IRC channel if you have any questions. Thank you for your patience.

UPDATE (16 June, 11 PM): All OCF services, including mail, webmail, and SSH logins for accounts starting with the letters q-z, should be working now. Please let us know if anything appears to be broken, and thank you for your patience!

UPDATE (17 June, 1 PM): Webmail has been fixed.

UPDATE (17 June, 1:30 PM): The OCF's main disk array is having problems — we've taken all OCF services down while we investigate.
diff --git a/docs/news/posts/blogger-post-2010-06-21-1.md b/docs/news/posts/blogger-post-2010-06-21-1.md new file mode 100644 index 0000000..3522126 --- /dev/null +++ b/docs/news/posts/blogger-post-2010-06-21-1.md @@ -0,0 +1,6 @@ +--- +date: 2010-06-21 +--- + +# Read-only home directories, mailboxes, and html directories +As you may know by now OCF services have been marginally restored and you can now access your files. However, you may have also noticed that you get strange errors, "Permission Denied", and are generally unable to edit your files. This is because we have mounted all your data Read-Only. This is a consequence of impending failure of our disk array, and we are in the process of migrating to a new one. Stay tuned for changes. diff --git a/docs/news/posts/blogger-post-2010-06-24-1.md b/docs/news/posts/blogger-post-2010-06-24-1.md new file mode 100644 index 0000000..692f37b --- /dev/null +++ b/docs/news/posts/blogger-post-2010-06-24-1.md @@ -0,0 +1,6 @@ +--- +date: 2010-06-24 +--- + +# More problems.... +There was an issue with our networking today, and a few of our servers responded very negatively to the loss of network connectivity. We are working to get this resolved asap, like before we don't have an ETA on how long it will take. We will keep you posted. diff --git a/docs/news/posts/blogger-post-2010-07-03-1.md b/docs/news/posts/blogger-post-2010-07-03-1.md new file mode 100644 index 0000000..beeb2be --- /dev/null +++ b/docs/news/posts/blogger-post-2010-07-03-1.md @@ -0,0 +1,6 @@ +--- +date: 2010-07-03 +--- + +# Further Updates +Thank you for all your patience, its really appreciated. Our new disk array should be back up, and we will slowly restore services after the July 4th break. I don't want to rush anything since its just going to make it harder to fix if something goes wrong.

I want to make sure everyone knows we have multiple goals when addressing the current problems. While we try to maximize our uptime, we are also concurrently rebuilding our current system. At the beginning of summer, we did send a downtime announcement that spanned the whole summer. At some point we may decide the current system is not worth salvaging, and focus our efforts on the rebuild.

Furthermore, we will shortly attempt to rebuild our core servers, which will take down our entire system. We will try our best to build ad hoc login servers, but anticipate more downtime in July. diff --git a/docs/news/posts/blogger-post-2010-07-13-1.md b/docs/news/posts/blogger-post-2010-07-13-1.md new file mode 100644 index 0000000..985456a --- /dev/null +++ b/docs/news/posts/blogger-post-2010-07-13-1.md @@ -0,0 +1,6 @@ +--- +date: 2010-07-13 +--- + +# Users can now write to home directories +The OCF has been reasonably stable for the past few days, I have decided to allow users to write to their home directories. diff --git a/docs/news/posts/blogger-post-2010-07-19-1.md b/docs/news/posts/blogger-post-2010-07-19-1.md new file mode 100644 index 0000000..054d267 --- /dev/null +++ b/docs/news/posts/blogger-post-2010-07-19-1.md @@ -0,0 +1,6 @@ +--- +date: 2010-07-19 +--- + +# Mysql Upgrade +Hi all,
We just upgraded our mysql server, let me know if anything doesn't work. Some of our client programs may be old, so if it says stuff like "Authentication protocol not supported" don't worry, its just that our clients are using old programs. diff --git a/docs/news/posts/blogger-post-2010-07-31-1.md b/docs/news/posts/blogger-post-2010-07-31-1.md new file mode 100644 index 0000000..e112ac3 --- /dev/null +++ b/docs/news/posts/blogger-post-2010-07-31-1.md @@ -0,0 +1,6 @@ +--- +date: 2010-07-31 +--- + +# Servicing our disk array +The OCF will likely be down the entire weekend as we update, error check, and test our disk array. Please check back later for more updates diff --git a/docs/news/posts/blogger-post-2010-08-13-1.md b/docs/news/posts/blogger-post-2010-08-13-1.md new file mode 100644 index 0000000..bafbcca --- /dev/null +++ b/docs/news/posts/blogger-post-2010-08-13-1.md @@ -0,0 +1,6 @@ +--- +date: 2010-08-13 +--- + +# Anticipate downtime this weekend +Hi all,
We will try to rebuild one of our main servers this weekend, anticipate some downtime and/or loss of functionality. diff --git a/docs/news/posts/blogger-post-2010-08-14-1.md b/docs/news/posts/blogger-post-2010-08-14-1.md new file mode 100644 index 0000000..43da9a5 --- /dev/null +++ b/docs/news/posts/blogger-post-2010-08-14-1.md @@ -0,0 +1,6 @@ +--- +date: 2010-08-14 +--- + +# Update +Hi all,
Some of you may have noticed your cgi scripts have broken, this is probably due to the fact that our new webserver has a different directory structure, and your scripts rely on a hard coded directory path. Don't worry this is only temporary. I used different nfs mount points to make mounting user data easier while we were swapping ip's around. It should be resolved when the server is on its proper hardware. diff --git a/docs/news/posts/blogger-post-2010-08-21-1.md b/docs/news/posts/blogger-post-2010-08-21-1.md new file mode 100644 index 0000000..2449f13 --- /dev/null +++ b/docs/news/posts/blogger-post-2010-08-21-1.md @@ -0,0 +1,6 @@ +--- +date: 2010-08-21 +--- + +# Webmail Restored +We've restored webmail functionality. Please let us know if there are any problems or hiccups. diff --git a/docs/news/posts/blogger-post-2010-09-02-1.md b/docs/news/posts/blogger-post-2010-09-02-1.md new file mode 100644 index 0000000..39ad39f --- /dev/null +++ b/docs/news/posts/blogger-post-2010-09-02-1.md @@ -0,0 +1,6 @@ +--- +date: 2010-09-02 +--- + +# Webserver down for some maintenance +We are moving the webserver onto some better hardware, it should be a bit faster after the transition... diff --git a/docs/news/posts/blogger-post-2010-10-10-1.md b/docs/news/posts/blogger-post-2010-10-10-1.md new file mode 100644 index 0000000..5d65fcd --- /dev/null +++ b/docs/news/posts/blogger-post-2010-10-10-1.md @@ -0,0 +1,6 @@ +--- +date: 2010-10-10 +--- + +# Expected downtime +Hi all,
Yesterday's reboot left behind some nasty problems in our mail volume, we are going to take some downtime to fix it. 7pm-11pm, I expect system-wide downtime, I apologize for whatever inconvenience this has caused. In hindsight we should have handled this better, but we learn from our mistakes.

Sorry for the trouble.

Update 4:56pm
We may start working a bit early at around 5:30-6:00 so don't be surprised if /var/mail disappears during that time.

Update 8:40pm
Stuff should be working again for the most part, thanks for your patience diff --git a/docs/news/posts/blogger-post-2010-10-10-2.md b/docs/news/posts/blogger-post-2010-10-10-2.md new file mode 100644 index 0000000..d8d55ee --- /dev/null +++ b/docs/news/posts/blogger-post-2010-10-10-2.md @@ -0,0 +1,6 @@ +--- +date: 2010-10-10 +--- + +# Disk array reboot +Hi all,
We rebooted our disk array, resulting about 30 mins of downtime system wide. Just notifying all in case there are any stray stale nfs handles hanging around, or if any scripts broke. Don't panic if you couldn't log into our machines during this time, it was nothing serious. diff --git a/docs/news/posts/blogger-post-2010-10-20-1.md b/docs/news/posts/blogger-post-2010-10-20-1.md new file mode 100644 index 0000000..c10b2c0 --- /dev/null +++ b/docs/news/posts/blogger-post-2010-10-20-1.md @@ -0,0 +1,6 @@ +--- +date: 2010-10-20 +--- + +# Mysql Down +Mysql went down for a short period of time yesterday, we hosted the service from backups in a read-only mode for the night.

UPDATE (15:41):
ETA of 8-10hrs before we get mysql up and running again.

UPDATE (19:41):
mysql, postgresql back in business. props to jaws.ocf.berkeley.edu, for performing admirably diff --git a/docs/news/posts/blogger-post-2010-10-21-1.md b/docs/news/posts/blogger-post-2010-10-21-1.md new file mode 100644 index 0000000..0ed25f6 --- /dev/null +++ b/docs/news/posts/blogger-post-2010-10-21-1.md @@ -0,0 +1,6 @@ +--- +date: 2010-10-21 +--- + +# Other failures +In a rather unlucky streak of timing here are other known failures in the OCF (today was not the best day).

2 printers (1 critically)
3 infrastructure related servers

Will update you as we get things fixed. diff --git a/docs/news/posts/blogger-post-2010-11-10-1.md b/docs/news/posts/blogger-post-2010-11-10-1.md new file mode 100644 index 0000000..9b575e6 --- /dev/null +++ b/docs/news/posts/blogger-post-2010-11-10-1.md @@ -0,0 +1,6 @@ +--- +date: 2010-11-10 +--- + +# ASUC Building Power Outage +Hi,
Tonight from 2am-6am Esheleman Hall will have a power outage, stay tuned for updates.

Update 8:44: Everything except the OCF webserver/disk array/authentication servers has now been shutdown.

Update 9:00pm Shutting down mysql

Update: 9:15pm Webserver shutdown

Update: 9:20pm Disk array shutdown/authentication servers shut down. The only things that are still up are infrastructure related servers. These allow us to manage machines remotely. The UPS says 2:15 of runtime, the outage is 4hrs, lets hope 4hrs is a conservative estimate?

Update 6:36am Marginally restored our infrastructure, we are running tests to make sure everything that is up is working. Login Servers will be up shortly

Update 7:04am There are a few issues coming back up we are looking to get them resolved ASAP

Update 7:31am We had some disk array issues, they seemed to be resolved for the time being, all the windows machines work, printing works, and we will soon boot up the login servers after we are sure permissions and such are working properly

Update 7:56 Our DNS server is being stubborn, seems to be the root cause of recent issues. Mysql should be back up

Update 8:04am FSCK time, what a fun way to start the moring, fsck'ing broken filesystems

Update 8:12am The webserver should be working again

Update 8:22am DNS is plodding along, expect a delay between 9-10:30 since I have class at this time.

Update 8:41am FSCK on the DNS server, will likely be down for a while, login servers should be up and running, docs and webmail should work too.

Update 8:44am spoke too soon disregard the previous post

Update 10:56am still working on getting dns up.

Update 11:11am DNS should be up now

Update 6:40pm Reaching hour 30 of this adventure, most of our services have been restored. Mail is a work in progress, but your stored email should be fully accessible now. apocalypse.ocf.berkeley.edu doesn't seem to turn on, so we will keep that off for now (while we straighten out everything else). diff --git a/docs/news/posts/blogger-post-2010-11-17-1.md b/docs/news/posts/blogger-post-2010-11-17-1.md new file mode 100644 index 0000000..aeaa80f --- /dev/null +++ b/docs/news/posts/blogger-post-2010-11-17-1.md @@ -0,0 +1,6 @@ +--- +date: 2010-11-17 +--- + +# Scheduled downtime this weekend +
Hey OCFers,

In order to give OCF mail a much-needed reliability boost, we plan to incrementally replace our current mail infrastructure with a centralized, redundant new system. The current system is scattered and full of single points of failure — as an example, our SpamAssassin server didn't survive last week's power outage, and our mail servers were forced to reject all incoming mail until we were able to build a replacement spam filter on a different machine. Our hope is that this new system will be significantly less volatile and more maintainable.

To that end, we'll be taking OCF mail down this Saturday. We'll be migrating mail storage, outgoing SMTP, IMAP/POP, and webmail to the new mail server. Incoming mail will not be accepted while we're working on the servers, but we'll try to keep read-only access to your old mail online for most of the day. As always, you can follow our progress here and on IRC.

UPDATE (20 November, 12:15PM): It begins! We've taken incoming mail offline and made /var/mail read-only for the mailspool migration.

UPDATE (20 Nov, 2:30PM): We've finished copying data from the old mailstore to the new one and are in the process of switching NFS servers. IMAP/POP access to mail is now down.

UPDATE (20 Nov, 5:30PM): We're having a bit of trouble with the new NFS server; IMAP/POP is still down, and our login servers hung and had to be rebooted a short while ago. tsunami and conquest should be working now, though.

UPDATE (20 Nov, 6:45PM): The NFS server is now up and running, now that we've squashed a pesky NFSv4-related bug (thanks to sluo and dwc for their help!). /var/mail is no longer read-only. Once we finish migrating disk quota information, we'll bring the mail servers back online. Thanks for your patience.

UPDATE (21 Nov, 12:15AM): /var/mail has been fully migrated to the new NFS server, and we've re-enabled the old mail servers. Incoming mail and POP/IMAP still live on the old infrastructure, but now that the NFS server migration is complete we should be able to set up the new infrastructure with minimal downtime.

To be clear, we have not yet migrated SMTP, IMAP/POP, or webmail to the new mail server, but we plan to do so in the near future. Watch this space for updates.
diff --git a/docs/news/posts/blogger-post-2010-11-23-1.md b/docs/news/posts/blogger-post-2010-11-23-1.md new file mode 100644 index 0000000..8214996 --- /dev/null +++ b/docs/news/posts/blogger-post-2010-11-23-1.md @@ -0,0 +1,6 @@ +--- +date: 2010-11-23 +--- + +# Unexpected Downtime +The OCF's DNS server just underwent spontaneous massive existence failure and we are trying to get the system to boot. Further bulletins as events warrant.

In other news, there was a hardware failure of unknown origin on the new mail server and /var/mail was down for a few minutes while we recovered the RAID array. Sorry for the inconvenience.

Update(10:30PM): DNS is back up. Sorry for the inconvenience again.

Update(11:00PM): The login servers had the blues during the DNS outage. They have been rebooted, and are fully functional again.
diff --git a/docs/news/posts/blogger-post-2010-12-10-1.md b/docs/news/posts/blogger-post-2010-12-10-1.md new file mode 100644 index 0000000..90cf498 --- /dev/null +++ b/docs/news/posts/blogger-post-2010-12-10-1.md @@ -0,0 +1,6 @@ +--- +date: 2010-12-10 +--- + +# OCF Mail Down +OCF mail is down; please stay tuned for updates.

UPDATE (10 December): The server which hosts /var/mail had a drive failure (one of the pins on the boot drive was bent, which short-circuited something and caused a system shutdown). We've restored the server using its other drive and are rebuilding RAID. All mail services should now be functional.

UPDATE (11 December): The mailspool server is down again, and its remote management system seems to be malfunctioning. Mail is down until an OCF staffer living in Berkeley can investigate; if the server is irreparably damaged, we'll have to wait until finals are over to rebuild it. Sorry for the inconvenience.

UPDATE (12 December): Some of you have expressed concern that your data may have been lost. Sorry for any alarm — rest assured, your data is safe, and we do have recent backups of /var/mail.

As part of our transition to a new mail infrastructure, we recently migrated /var/mail from our central disk array to a new machine with a better NFS server. We didn't realize until the server shut down two days ago that one of its drives was damaged, at which point we swapped out the drive and turned the machine on again. The server shut down again yesterday, and S.M.A.R.T. is predicting another drive failure, so we fear the motherboard or drive controller may have been damaged by the faulty drive. The server runs and your data is intact, but rather than just rebuilding RAID again and risking a major meltdown during finals week, we've decided to leave everything off after finals, when we'll have time to take care of this properly.

UPDATE (14 December): As of early morning today, mail is tentatively operational. Last night we transitioned /var/mail to an alternate hard disk in a temporary server. Maintenance work will be scheduled soon, especially over break. Special thanks to the ASUC for their cooperation.
diff --git a/docs/news/posts/blogger-post-2011-01-17-1.md b/docs/news/posts/blogger-post-2011-01-17-1.md new file mode 100644 index 0000000..e4135e6 --- /dev/null +++ b/docs/news/posts/blogger-post-2011-01-17-1.md @@ -0,0 +1,6 @@ +--- +date: 2011-01-17 +--- + +# Mail scheduled maintenance +As part of our ongoing consolidation and maintenance of mail infrastructure, OCF mail will be scheduled for maintenance in short period(s) during the beginning of the spring semester.

We apologize for the inconvenience and hope for better performance, reliability, and redundancy in the long run.

Mail is currently scheduled for maintenance on Monday, January 17, while /var/mail is migrated to a permanent mail server. Incoming mail will not be accepted during this time, but read-only access will be available as much as possible through the login servers (but not on POP/IMAP and webmail).

As always, you can follow our progress here and on IRC.

UPDATE 05:25p: Maintenance will begin shortly. /var/mail will now be mounted read-only.

UPDATE 06:00p: /var/mail is currently being migrated to a permanent server. It will still be mounted read-only, and accessible through the login servers (but not on POP/IMAP and webmail).

UPDATE Tuesday 01:40p: Mail should now be back up, including the login servers, POP/IMAP, and webmail. diff --git a/docs/news/posts/blogger-post-2011-01-19-1.md b/docs/news/posts/blogger-post-2011-01-19-1.md new file mode 100644 index 0000000..99124bc --- /dev/null +++ b/docs/news/posts/blogger-post-2011-01-19-1.md @@ -0,0 +1,6 @@ +--- +date: 2011-01-19 +--- + +# OCF Not Open Until Furthur Notice +Due to unexpected problems with the Windows Server, the Windows lab machines do not work.
We will update you as progress happens.

Update 01/19 at 09:17 PM: We were able to implement a temporary solution so that some of the Windows computers in the lab work (as well as some of the Linux computers). The computers with blue post-it notes definitely do not work. The others worked when last checked. We apologize for the inconvenience.

Update 01/20: The machines previously with blue post-it notes are now working. All computers should be working as before. diff --git a/docs/news/posts/blogger-post-2011-02-10-1.md b/docs/news/posts/blogger-post-2011-02-10-1.md new file mode 100644 index 0000000..516f247 --- /dev/null +++ b/docs/news/posts/blogger-post-2011-02-10-1.md @@ -0,0 +1,6 @@ +--- +date: 2011-02-10 +--- + +# Login servers +We are taking apocalypse down for maintenance. Most users should not be affected, however SSHing directly into apocalypse.ocf.berkeley.edu will not work.

To SSH into a login server, use the hostname ocf.berkeley.edu, as specified by our documentation.

Tuesday outage

On an unrelated note, tsunami (our primary login server) was down between 7:00am and 2:40pm on Tuesday, Feb 8. We apologize for the inconvenience.

Update: Tuesday, February 22

apocalypse.OCF has been permanently discontinued as a SPARC login server. We will be discontinuing support for the SPARC architecture.

tsunami is currently our 32-bit x86 login server, and was upgraded on Sunday to Debian 6.0.

diff --git a/docs/news/posts/blogger-post-2011-03-02-1.md b/docs/news/posts/blogger-post-2011-03-02-1.md new file mode 100644 index 0000000..fddd591 --- /dev/null +++ b/docs/news/posts/blogger-post-2011-03-02-1.md @@ -0,0 +1,6 @@ +--- +date: 2011-03-02 +--- + +# Print quotas +Our print server crashed a week ago and users' print quotas were wiped. A Board of Directors resolution was passed to set each users' quota to 175 pages given that we are in the 7th week of the semester. Users can email staff and will be dealt with on a case-by-case basis. diff --git a/docs/news/posts/blogger-post-2011-03-24-1.md b/docs/news/posts/blogger-post-2011-03-24-1.md new file mode 100644 index 0000000..31151a0 --- /dev/null +++ b/docs/news/posts/blogger-post-2011-03-24-1.md @@ -0,0 +1,6 @@ +--- +date: 2011-03-24 +--- + +# flood.OCF rebuild +flood.OCF has been taken offline as the machine is being rebuilt. Users should see no effect as all services running on flood have been transitioned to another server. diff --git a/docs/news/posts/blogger-post-2011-03-30-1.md b/docs/news/posts/blogger-post-2011-03-30-1.md new file mode 100644 index 0000000..b282067 --- /dev/null +++ b/docs/news/posts/blogger-post-2011-03-30-1.md @@ -0,0 +1,6 @@ +--- +date: 2011-03-30 +--- + +# Websites down +Our webserver ran into troubles with virtualization and crashed at 10:15pm last night. We are working to bring it back up and apologize for the inconvenience. User data should not been affected and is still accessible over SSH.

Update 11:07am: Should be back up. diff --git a/docs/news/posts/blogger-post-2011-04-05-1.md b/docs/news/posts/blogger-post-2011-04-05-1.md new file mode 100644 index 0000000..50ab644 --- /dev/null +++ b/docs/news/posts/blogger-post-2011-04-05-1.md @@ -0,0 +1,6 @@ +--- +date: 2011-04-05 +--- + +# Web Server Crash +The web server (death/www.OCF) had crashed again. We apologize for any inconvenience this may have caused.

UPDATE 12:03am: Fixed, web server is functioning again. diff --git a/docs/news/posts/blogger-post-2011-04-15-1.md b/docs/news/posts/blogger-post-2011-04-15-1.md new file mode 100644 index 0000000..d429515 --- /dev/null +++ b/docs/news/posts/blogger-post-2011-04-15-1.md @@ -0,0 +1,6 @@ +--- +date: 2011-04-15 +--- + +# Mail Server Down +The mail server is experiencing some issues and is currently down. We apologize for any inconvenience this may have caused. diff --git a/docs/news/posts/blogger-post-2011-04-17-1.md b/docs/news/posts/blogger-post-2011-04-17-1.md new file mode 100644 index 0000000..9371bfa --- /dev/null +++ b/docs/news/posts/blogger-post-2011-04-17-1.md @@ -0,0 +1,6 @@ +--- +date: 2011-04-17 +--- + +# Mail server rebuilt +As the previous post notes, our primary mail server (war.OCF aka mail.OCF) was down on Friday and parts of Thursday and Saturday.

At around 3:30a Thursday the server began to slow down on processing mail, and by 3:15p Friday the server had crashed. We could not turn the server back on, suspecting file corruption on the SPARC hypervisor the Solaris LDOM runs on.

We began rebuilding the mail server with Debian 6 on physical x86 hardware Friday night, and after 9am Saturday services were coming up. IMAP, POP, SMTP, and SMTPAUTH should all be up as of 12:30p Saturday.

We're using a new SSL certificate on the new mail server.

Since we're still running parts of mail on a mix of other machines running Solaris and Debian 5, this is a temporary solution. Maintenance work should be expected soon. We apologize for the inconvenience.

As always, you can email any suggestions or comments to staff@OCF.Berkeley.EDU.

UPDATE Sat 5:15p: Our incoming mail server (sandstorm.OCF) has a backlog of emails to process. There may be delays in receiving emails.

UPDATE Sat 11:45p: special local (slocal) mail delivery mail processing is no longer supported, please update your .forward files accordingly.

UPDATE Tues 11:55p: We've had difficulties with the incoming mail server, so mail may have been temporarily (or possibly in some rare cases, permanently rejected) when sent to an OCF address. We've replaced the Solaris machine with the newly-rebuilt mail server. We apologize for the inconvenience, and hope that there will be no bugs. diff --git a/docs/news/posts/blogger-post-2011-04-24-1.md b/docs/news/posts/blogger-post-2011-04-24-1.md new file mode 100644 index 0000000..48ad4d3 --- /dev/null +++ b/docs/news/posts/blogger-post-2011-04-24-1.md @@ -0,0 +1,6 @@ +--- +date: 2011-04-24 +--- + +# OCF internet connection down +The campus router which provides our Internet connection went down at 5:40am. All services are down.

UPDATE 5:45am: Router is online. Services should be up again. diff --git a/docs/news/posts/blogger-post-2011-05-03-1.md b/docs/news/posts/blogger-post-2011-05-03-1.md new file mode 100644 index 0000000..e372c39 --- /dev/null +++ b/docs/news/posts/blogger-post-2011-05-03-1.md @@ -0,0 +1,6 @@ +--- +date: 2011-05-03 +--- + +# Login server, webmail, and wiki down +The login server (tsunami aka ssh.OCF), webmail, and wiki are not accessible since 1:01pm today.

These services are provided by virtual machines. The physical hypervisor in which the virtual machines are run has issues accessing its data store, and we are investigating SCSI failure with the hard drives and controller.

No user data is stored on these machines, so your data is not affected. We apologize for the inconvenience.

If you need SFTP/SSH access, you may temporarily connect to one of the desktops, pileup.OCF.Berkeley.EDU, instead of connecting to ssh.OCF.Berkeley.EDU which is currently unavailable.

UPDATE 6:45p: The login server (SFTP/SSH) is back up.

UPDATE 6:52p: Webmail and wiki are back up. diff --git a/docs/news/posts/blogger-post-2011-05-12-1.md b/docs/news/posts/blogger-post-2011-05-12-1.md new file mode 100644 index 0000000..4722f70 --- /dev/null +++ b/docs/news/posts/blogger-post-2011-05-12-1.md @@ -0,0 +1,6 @@ +--- +date: 2011-05-12 +--- + +# Off-peak mail maintenance +The mail server was taken down from 04:39a to 05:55a today for maintenance and cleanup. SSH and SFTP logins may have failed during that time. All services should now be operational. diff --git a/docs/news/posts/blogger-post-2011-06-01-1.md b/docs/news/posts/blogger-post-2011-06-01-1.md new file mode 100644 index 0000000..c5470c8 --- /dev/null +++ b/docs/news/posts/blogger-post-2011-06-01-1.md @@ -0,0 +1,6 @@ +--- +date: 2011-06-01 +--- + +# Issues sending mail through webmail and webserver +It was discovered that users could not send mail through webmail or the webserver for the past month.

If you sent an email from OCF webmail between March 25 and April 31, please confirm that it was actually received.

You will not have received emails from your website in the past 4 weeks.

These issues have now been fixed. We apologize for any inconvenience and for the delay in discovering them. diff --git a/docs/news/posts/blogger-post-2011-06-05-1.md b/docs/news/posts/blogger-post-2011-06-05-1.md new file mode 100644 index 0000000..c91dd3d --- /dev/null +++ b/docs/news/posts/blogger-post-2011-06-05-1.md @@ -0,0 +1,6 @@ +--- +date: 2011-06-05 +--- + +# Internet outage +The gateway which our uplink is connected to was offline during the following periods this morning:

06:41a - 06:43a
06:50a - 06:56a
07:06a - 07:43a

The OCF was not externally accessible during this time. Since the gateway is operated by the campus IST department, we cannot verify the cause of the outage.

We suspect this was related to a planned power outage at the IST data center, although the campus network which we are connected to was not planned to be affected. diff --git a/docs/news/posts/blogger-post-2011-06-13-1.md b/docs/news/posts/blogger-post-2011-06-13-1.md new file mode 100644 index 0000000..e5cf436 --- /dev/null +++ b/docs/news/posts/blogger-post-2011-06-13-1.md @@ -0,0 +1,10 @@ +--- +date: 2011-06-13 +--- + +# Login server and wiki downtime +Secure shell login and access to the wiki were unavailable Monday morning and part of the afternoon. These services are hosted on virtual machines whose hypervisor (once again) had problems accessing its SCSI drives. The machine was reset, and is running normally again. +
+
Additionally, the switch connecting the alternate login machine (pileup) to the OCF had been switched off. This has also been remedied. +
+
As was the case last time, no user data is stored on these machines, so your data is not affected. We apologize for the inconvenience. diff --git a/docs/news/posts/blogger-post-2011-06-25-1.md b/docs/news/posts/blogger-post-2011-06-25-1.md new file mode 100644 index 0000000..448c0f9 --- /dev/null +++ b/docs/news/posts/blogger-post-2011-06-25-1.md @@ -0,0 +1,6 @@ +--- +date: 2011-06-25 +--- + +# Service downtime +There was intermittent down time between 11a-12:30p PDT due to software updates. diff --git a/docs/news/posts/blogger-post-2011-08-15-1.md b/docs/news/posts/blogger-post-2011-08-15-1.md new file mode 100644 index 0000000..a9e7ab4 --- /dev/null +++ b/docs/news/posts/blogger-post-2011-08-15-1.md @@ -0,0 +1,6 @@ +--- +date: 2011-08-15 +--- + +# Login server restarted +The login server (tsunami) was restarted at 10:14 PM for unanticipated maintenance. Zabbix, an uptime monitoring software, was not communicating with tsunami properly. The iptables firewall were flushed, causing network connectivity to be disrupted briefly. We apologize for the inconvenience. diff --git a/docs/news/posts/blogger-post-2011-08-18-1.md b/docs/news/posts/blogger-post-2011-08-18-1.md new file mode 100644 index 0000000..b7bee1a --- /dev/null +++ b/docs/news/posts/blogger-post-2011-08-18-1.md @@ -0,0 +1,8 @@ +--- +date: 2011-08-18 +--- + +# Service downtime +User information hosted in LDAP was unavailable from 4p to 9p PDT. Web hosting, email, and SSH/SFTP services were intermittently interrupted. Mail service was also taken down from 10p to 11p PDT for maintenance. We apologize for the inconvenience. +
+
Although no user data was affected, some email messages sent to OCF accounts during the service interruption may have been rejected and bounced to the sender. diff --git a/docs/news/posts/blogger-post-2011-08-19-1.md b/docs/news/posts/blogger-post-2011-08-19-1.md new file mode 100644 index 0000000..143b03e --- /dev/null +++ b/docs/news/posts/blogger-post-2011-08-19-1.md @@ -0,0 +1,6 @@ +--- +date: 2011-08-19 +--- + +# MySQL downtime +MySQL was unavailable between 9:08p and 1:52a for unanticipated maintenance. diff --git a/docs/news/posts/blogger-post-2011-09-11-1.md b/docs/news/posts/blogger-post-2011-09-11-1.md new file mode 100644 index 0000000..55fa5e7 --- /dev/null +++ b/docs/news/posts/blogger-post-2011-09-11-1.md @@ -0,0 +1,6 @@ +--- +date: 2011-09-11 +--- + +# Login server restarted +The primary login server (tsunami, ssh.OCF) was restarted at 4:00am PDT to apply a kernel security update as part of scheduled maintenance. diff --git a/docs/news/posts/blogger-post-2011-09-18-1.md b/docs/news/posts/blogger-post-2011-09-18-1.md new file mode 100644 index 0000000..f21ce83 --- /dev/null +++ b/docs/news/posts/blogger-post-2011-09-18-1.md @@ -0,0 +1,6 @@ +--- +date: 2011-09-18 +--- + +# Unanticipated Downtime +Due to some unanticipated network issues on the nfs server, all servers that depend on nfs are down. We apologize for the inconvenience.

Update 1:40am: All services have been restored on all servers.
Update 11:20am: An unrelated issue was denying IMAP/POP/SMTP authentication, fixed. diff --git a/docs/news/posts/blogger-post-2011-09-19-1.md b/docs/news/posts/blogger-post-2011-09-19-1.md new file mode 100644 index 0000000..7c84453 --- /dev/null +++ b/docs/news/posts/blogger-post-2011-09-19-1.md @@ -0,0 +1,6 @@ +--- +date: 2011-09-19 +--- + +# Some incoming mail rejected +Between September 1 and September 15, some incoming mail was incorrectly rejected as spam. A blacklist incorrectly included all mail servers, raising the score of incoming messages, and causing some high-scoring messages to be rejected at the SMTP stage. See Debian bug #641227.

Since these messages were rejected before their contents could be accepted, they cannot be recovered.

Email senders should have received a non-delivery report (bounce message) that stated:

Client host rejected: Mail appeared to be SPAM or forged. Ask your Mail/DNS-Administrator to correct HELO and DNS MX settins or to get removed from DNSBLs;


or

Client host rejected: temporarily blocked because of previous errors - retrying too fast.


We apologize for the inconvenience. diff --git a/docs/news/posts/blogger-post-2011-09-21-1.md b/docs/news/posts/blogger-post-2011-09-21-1.md new file mode 100644 index 0000000..3d33401 --- /dev/null +++ b/docs/news/posts/blogger-post-2011-09-21-1.md @@ -0,0 +1,6 @@ +--- +date: 2011-09-21 +--- + +# Unexpected downtime, RAID failure +Taking precautionary measures after RAID failure on our main disk array, we have taken down all services that depend on NFS, including web hosting, mail service, and home directory file access.

We are doing our best by working overnight so that service will be restored as soon as possible.

We apologize for the inconvenience.

Update Thurs Sep 22 05:45am


First, we want to apologize (again!) for the delay. We were hoping to restore service yesterday morning. Things didn't exactly work out...

While we don't want to blame the delay on excuses, we would also like to be straightforward about what is going on.

Last Sunday, our backup server began to fail, and by Tuesday, the largest volume, which contained backups of home and web directories, appeared to be unrecoverable. Data is stored in a RAID volume (meaning it is resilient to a certain number of hard drive failures), so the simultaneous hard drive failures/corruption that would have been required suggest hardware problems with the server itself and not (or not only) the hard drives. To be honest, hard drive or RAID controller failure is not completely unexpected for an aging machine with aging hard drives (the server easily predates all of current staff, so we don't know exactly how old). In 20/20 hindsight, we could have acquired and set up new hardware, but for a machine that hosts backup copies of data and is not directly accessible, the extra expense in time and money did not appear to be worthwhile.

On Tuesday at 10:30pm, the main disk array, which exports most data stored in NFS (home and web directories, and mail folders but not mail inboxes). Because RAID adds redundancy, no data was lost, but redundancy was lost, meaning future failures result in data loss. This is why we then took down all services that can access or edit this data and modified others (like printing) to not depend on it (this will however prevent you from being able to see your print quota, you will need to ask a staff member).

The disk array is in its third year of service, so hardware problems with the server itself are not really expected but not improbable either. However, even the most reliable hard drives, accessed constantly 24/7, can fail. We did not have any hard drives other than those in the disk array with capacities greater than or equal to 1 TB, meaning we could not immediately begin rebuilding the RAID volume (again, in hindsight, this was probably a mistake on our part), let alone any (more expensive) "enterprise-class" 1 TB hard drives as would be proper for (and which are currently used in) the disk array.

On Wednesday morning, we bought a temporary "desktop-class" hard drive. When we mounted the new hard drive in the disk array, it was detected but unrecognized and marked as "bad" on reboot. We tried unsuccessfully to work around the problem. Other hard drives (of smaller capacity; they cannot be used to rebuild the RAID volume) were recognized and usable for other purposes without errors. It seems highly unlikely that a brand new hard drive would be bad, and we could not find any sign of errors when testing and running diagnostics on the hard drive in other machines, so the disk array is suspect, but since it may work with other hard drives, is not clearly at fault either. (There appear to be firmware restrictions on intermixing "desktop class" and "enterprise class" drives.)

On Wednesday evening, as another precautionary measure, we planned out a procedure to replicate the data on other machines so that if another hard drive or the disk array were to fail, we would not have data loss or corruption. To prioritize, we are copying data in alphabetical order from enabled (i.e., not disabled) accounts to another hard drive on the disk array. We will remove this hard drive when done for safekeeping, and also copy the same data over our internal network to another server with a RAID 1 (mirror) setup.

We will try our best to restore service as soon as possible. We don't want to sound deceiving by suggesting a time earlier than what might end up happening, especially since we need to first make sure that existing data is safely backed up. Service downtime through the weekend is not acceptable but it is possible, and depending on any obstacles encountered, the length of downtime could be longer or shorter.

Our Board of Directors (comprised of interested OCF members, volunteer staff and "users" alike) currently meets weekly on Thursday at 6:45pm in the OCF lab. Our next meeting is today, and if you have any advice or comments, related or not to the downtime, we encourage you to attend.

Update Sat Sep 24 06:30pm


The ASUC Auxiliary is closed on weekends, and as a result we won't be able to obtain the package of hard drive replacements that we ordered until Monday, at which point we will be able to rebuild the array and bring services back online. We may be able to mount the disk array read-only before then if the local copy is complete.

Update Sun Sep 25 12:00am


The local copying of non-disabled accounts that was started on Thursday morning is about 90% finished. We're expecting it to be finished by the morning.

Update Sun Sep 25 10:30am


The login and mail servers are now mounting home directories read-only. SSH/SFTP will give you read-only access to your files, IMAP/POP/mutt/webmail will give you read-only access to your mail.

Unfortunately, the ASUC Auxiliary is closed on weekends, and as a result we won't be able to obtain the package of hard drive replacements that we ordered until Monday, at which point we will be able to repair the array and bring all services back up.

Update Sun Sep 25 11:30am


The web server is now serving web pages read-only. This may break some sites that require writing to the home or web directory. For the time being, you can optionally use our error message to give an HTTP 503 Service temporarily unavailable error with an explanation.

Update Mon Sep 26 02:30pm


We obtained and added the new hard drives to the array at 9am this morning. If there are no errors, we expect the resync to be complete by 5pm. We will then mount home directories with full read and write access in the state they were originally.


Update Mon Sep 26 05:15pm


Finally, all services are operational as before. This will hopefully be the last update... diff --git a/docs/news/posts/blogger-post-2011-10-10-1.md b/docs/news/posts/blogger-post-2011-10-10-1.md new file mode 100644 index 0000000..ab61a9f --- /dev/null +++ b/docs/news/posts/blogger-post-2011-10-10-1.md @@ -0,0 +1,6 @@ +--- +date: 2011-10-10 +--- + +# Mail service downtime +Mail service, including SMTP, IMAP, and POP, was unavailable between 2:19am and 3:06am because of transient errors with LDAP (user database). Some mail sent or received during this time may have been incorrectly rejected. diff --git a/docs/news/posts/blogger-post-2011-11-01-1.md b/docs/news/posts/blogger-post-2011-11-01-1.md new file mode 100644 index 0000000..ea7e27b --- /dev/null +++ b/docs/news/posts/blogger-post-2011-11-01-1.md @@ -0,0 +1,6 @@ +--- +date: 2011-11-01 +--- + +# Networking maintenance +There will be one day of downtime during the days of November 18-20 for networking maintenance. The anticipated downtime should be much less than one day. The exact date will be confirmed as the day approaches. diff --git a/docs/news/posts/blogger-post-2011-11-18-1.md b/docs/news/posts/blogger-post-2011-11-18-1.md new file mode 100644 index 0000000..8412080 --- /dev/null +++ b/docs/news/posts/blogger-post-2011-11-18-1.md @@ -0,0 +1,6 @@ +--- +date: 2011-11-18 +--- + +# Networking maintenance downtime +Downtime is scheduled between 8pm and midnight tonight.

Update 4:00am


We replaced some wiring and networking equipment between 11:30pm and 01:15am. Most services were only briefly affected. Web, mail, SSH, and dependent services were temporarily moved to another switch during the rewiring.

We thank a generous donor for two new gigabit switches. diff --git a/docs/news/posts/blogger-post-2011-12-07-1.md b/docs/news/posts/blogger-post-2011-12-07-1.md new file mode 100644 index 0000000..62f9d0b --- /dev/null +++ b/docs/news/posts/blogger-post-2011-12-07-1.md @@ -0,0 +1,6 @@ +--- +date: 2011-12-07 +--- + +# Lab and printing services uncertain after Spring 2012 +
Dilbert.com


From: OCF announcements
To: All OCF members
Subject: OCF lab and printing ending after Spring 2012
Date: Wed, Dec 7, 2011 01:31p PST

Hello OCF members,

Last night OCF's volunteer staff unanimously decided to close lab services, including free printing, after Spring 2012.

OCF, as with other student groups in Eshleman Hall, will be moved out of the building in August before it is demolished as part of Bears Initiative. Until new space is obtained elsewhere on campus, we have been allocated temporary space in Hearst Gymnasium basement for the duration of Lower Sproul construction, which is projected to last for at least 3 years.

This area is limited both in space and technical resources, and we feel that it would be a disservice to all OCF members to provide an unreliable and inadequate level of service, in both quality and quantity. We would not be able to fit much of our computing and printing equipment into the space allotted to us, and time constraints limit what can be set up before the fall semester begins.

We understand and share your concern for the OCF. We too are students and members of the OCF. This is a drastic change for the organization, which has had a lab since 1989.

We want to restore the lab and its services as soon as possible, but we must also keep in mind the process of our last two moves (to Heller Lounge in the MLK Student Union, and to our present location in Eshleman). It would be dishonest if we were not to give notice of lab downtime, which has been unavoidable and
excessive during each move. Furthermore, the compressed space and needs of other student groups constrain us more than in the past.

We hope that by shifting resources, we will be able to compensate by expanding our other services, including currently web hosting, disk space, email, and shell accounts, and ensure that they would not be disrupted by a move as they have been before. We are looking at our options for negotiating server space and we will continue hosting websites, including those of student groups. We will better assist other students and student groups as a technical resource given our close proximity in the temporary space, and encourage you to make use of what we would be able to offer.

We will be considering all options, including other space if available, and appreciate any feedback or advice you can offer. We don't want to let you down...

Good luck on finals guys,
OCF staff members,
Eshleman basement

P.S. One of the printers is broken, so printing might be slower than we'd hope, but please, enjoy our lab while it lasts. 9am-9pm hours during dead week and finals, see our website for more info.

You have received this announcement to update you about important changes to the OCF. The Open Computing Facility is an all-volunteer student organization providing free shell accounts, disk space, web hosting, email, and printing from a lab, lounge, and server room in Eshelman Hall.


Update Dec 8: Daily Cal has published an article: Computing facility in Eshleman Hall to close after next semester

Update Dec 17: To clarify, OCF is not going away, but staff felt that, given our constraints, it would not be possible to guarantee a public lab next fall, and misleading if we did not inform members. The post's title has been updated to reflect that lab and printing services are uncertain, since no decision is made in stone, and the constraints which led to our decision can (and hopefully will) change. Sorry about the confusion.

Two years ago OCF completed its 15-month move from Heller Lounge in MLK to the larger space in Eshleman, with excessive lab and server downtime (caused in part by the discovery of asbestos and later a failed pump flooding the new lab, then a pipe burst dropping raw sewage, then a network and later power outage which was prolonged). The current move leaves us two weeks (Aug 1 to Aug 15) to move and bring the lab up in a much smaller space before the semester begins. That also includes getting rid of desktops, servers, and equipment that would not fit in the space, since Moore's Law makes the storage of aging computer hardware uneconomical.

We also hoped that by not committing ourselves to the lab immediately, we would leave ourselves time to ensure that non-lab services, like web hosting, would not be disrupted by the move. During the last move our web hosting, used by many student groups, was down for several months, and the complaints were not pretty. To this day we hear about OCF "unreliability" based on that downtime.

And lastly, we have every intent to restore our lab when it is possible and when we have the logistics of the space worked out. To quote an ex-staff member: "The group's choice to focus on back-end makes a lot of sense given the uncertainty with construction and space."

And yes, this makes us very unhappy. diff --git a/docs/news/posts/blogger-post-2011-12-17-1.md b/docs/news/posts/blogger-post-2011-12-17-1.md new file mode 100644 index 0000000..94598b3 --- /dev/null +++ b/docs/news/posts/blogger-post-2011-12-17-1.md @@ -0,0 +1,6 @@ +--- +date: 2011-12-17 +--- + +# Intermittent downtime for maintenance +On December 16, we rewired parts of our network between 5pm and 7pm, resulting in intermittent unavailability for OCF services. We believe that the end of the semester would be a less intrusive time to make changes, especially since the lab is closed until spring semester.

OCF, as with other student groups in Eshleman Hall, will be moved out of the building in August before it is demolished as part of Bears Initiative. We are aggressively consolidating our hardware (reducing the number of physical servers), since space for our hardware after the move will be limited, and working to prevent these changes from having an adverse effect on performance and reliability. We apologize for the inconvenience.

See also: Lab and printing services uncertain after Spring 2012 diff --git a/docs/news/posts/blogger-post-2012-01-26-1.md b/docs/news/posts/blogger-post-2012-01-26-1.md new file mode 100644 index 0000000..05b2399 --- /dev/null +++ b/docs/news/posts/blogger-post-2012-01-26-1.md @@ -0,0 +1,6 @@ +--- +date: 2012-01-26 +--- + +# Web Server Downtime +Tonight, the web server ran out of swap space and is currently experiencing downtime.

Since there are updates pending on the server, rebooting it runs the risk of it not booting up properly afterwards.

Work on the (much more powerful) virtual machine that will become the new web server began a week ago, and staff will meet this Saturday to finish it.

Update Jan 26 8pm: Fixed, but there may be downtime on Saturday as web hosting is migrated to a new server. diff --git a/docs/news/posts/blogger-post-2012-02-18-1.md b/docs/news/posts/blogger-post-2012-02-18-1.md new file mode 100644 index 0000000..8a7dc13 --- /dev/null +++ b/docs/news/posts/blogger-post-2012-02-18-1.md @@ -0,0 +1,13 @@ +--- +date: 2012-02-18 +--- + +# RAID Failure tonight +Hello OCF Users,
+
+Tonight our back-end file server had a RAID failure that initially looked like full swap space.  We hard-rebooted after a few minutes wait for the server to reboot itself.
+
+It's lucky that this RAID failure took place during off hours so we can begin maintenance immediately.  Expect a post again when the RAID issues are fully resolved.
+
+Update March 11: This RAID failure was on a drive that did not have user data on it. There are no problems at all. Sorry for the delayed post. This problem was solved the same day as the RAID failure.
+
diff --git a/docs/news/posts/blogger-post-2012-03-11-1.md b/docs/news/posts/blogger-post-2012-03-11-1.md new file mode 100644 index 0000000..aef5ef0 --- /dev/null +++ b/docs/news/posts/blogger-post-2012-03-11-1.md @@ -0,0 +1,8 @@ +--- +date: 2012-03-11 +--- + +# MySQL migration +MySQL is being migrated to a new server. All MySQL databases will be locked to prevent editing while the migration is happening. However, all data will still be in read-only mode. Once the migration is done, we will post an update. Thank you for your patience.
+
+Update 7:15 AM: The migration has been completed successfully. diff --git a/docs/news/posts/blogger-post-2012-05-23-1.md b/docs/news/posts/blogger-post-2012-05-23-1.md new file mode 100644 index 0000000..9bd9f1a --- /dev/null +++ b/docs/news/posts/blogger-post-2012-05-23-1.md @@ -0,0 +1,8 @@ +--- +date: 2012-05-23 +--- + +# Emergency maintenance of login server (ssh.ocf, tsunami) +Due to unforeseen issues with the server, we are taking it down for emergency maintenance. We will update you as progress happens. We apologize for the inconvenience.
+
+Update 4:30 PM: The maintenance has been completed successfully. Thank you for you patience. diff --git a/docs/news/posts/blogger-post-2012-06-11-1.md b/docs/news/posts/blogger-post-2012-06-11-1.md new file mode 100644 index 0000000..02f7c71 --- /dev/null +++ b/docs/news/posts/blogger-post-2012-06-11-1.md @@ -0,0 +1,8 @@ +--- +date: 2012-06-11 +--- + +# Emergency email maintenance (mail.OCF, imap.OCF, smtp.OCF, sandstorm.OCF) +Due to unforseen issues with our network file share on our email server, we have taken down email to troubleshoot this issue. We will update you as progress happens. We apologize for this inconvenience.
+
+Update 12:00AM: This work has been completed successfully. diff --git a/docs/news/posts/blogger-post-2012-06-12-1.md b/docs/news/posts/blogger-post-2012-06-12-1.md new file mode 100644 index 0000000..ee91e27 --- /dev/null +++ b/docs/news/posts/blogger-post-2012-06-12-1.md @@ -0,0 +1,70 @@ +--- +date: 2012-06-12 +--- + +# Web server replaced +
+

+ + + + + + +Web serving or: How We Learned to Stop Worrying and Love Debian

+The OCF web server, which hosts http://www.ocf.berkeley.edu and hundreds of virtual hosts for group accounts, was migrated over to our newest (and fastest) server running Debian GNU/Linux on June 9. + +
+

+ + + + + +The Good

+
+Websites, especially with dynamic content, should be loaded (much) faster. Features might be added later down the road.
+

+ + + + + + +The Bad

+This is possibly our most visible non-lab-related (intentional) change in the past few years, and when that happens, stuff tends to break. We made the change over summer to allow us the most time to fix things with minimal disruption, and before we are moved out of the basement of Eshleman Hall (sometime during or after August). + +
+

+ + + + + + +The Ugly

+It might take us a while to fix things. Contact us if something doesn't work that worked before. + +
+

+ + + + + + +Known unresolved issues

+ +

+ + +Troubleshooting

+ + +
diff --git a/docs/news/posts/blogger-post-2012-07-02-1.md b/docs/news/posts/blogger-post-2012-07-02-1.md new file mode 100644 index 0000000..dbe1e6c --- /dev/null +++ b/docs/news/posts/blogger-post-2012-07-02-1.md @@ -0,0 +1,11 @@ +--- +date: 2012-07-02 +--- + +# Scheduled webserver downtime +
+We are scheduling downtime on our primary server (hal) to physically inspect the machine between 6:30pm and 7:30pm July 1st. Notably, this will affect web hosting, MySQL, IRC, and wiki. We will again schedule downtime in one or two weeks to add two additional processors to hal.
+
+Update 6:47pm:  hal is being taken offline.
+Update 7:25pm:  hal and all services except IRC are back.
+Update 7:32pm:  All services operating normally.
diff --git a/docs/news/posts/blogger-post-2012-07-06-1.md b/docs/news/posts/blogger-post-2012-07-06-1.md new file mode 100644 index 0000000..114fe57 --- /dev/null +++ b/docs/news/posts/blogger-post-2012-07-06-1.md @@ -0,0 +1,15 @@ +--- +date: 2012-07-06 +--- + +# Scheduled webserver downtime again +
+
+We are (again) scheduling downtime on our primary server (hal) between 9:00pm and 10:30pm July 5st, to add two new processors. Notably, this will affect web hosting, MySQL, IRC, and wiki.
+Update 9:15pm: Services going down.
+ + + +
Mid-downtime action shot!
+(via Kenny Do)
+Update 10:24pm: Services are back up.
diff --git a/docs/news/posts/blogger-post-2012-07-22-1.md b/docs/news/posts/blogger-post-2012-07-22-1.md new file mode 100644 index 0000000..f4b587e --- /dev/null +++ b/docs/news/posts/blogger-post-2012-07-22-1.md @@ -0,0 +1,9 @@ +--- +date: 2012-07-22 +--- + +# Mail maintenance +
+Mail services (IMAP/POP/SMTP/webmail) are going down as we transition the mail server to a virtual machine. By doing so, we hope to keep mail available on a temporary server during our move out of Eshleman in August, since virtual machines (guests) can be migrated on-the-fly between physical hypervisors (hosts).
+
+Update 02:04p: All services back up. Mail spool (inbox files) will be migrated later.
diff --git a/docs/news/posts/blogger-post-2012-07-24-1.md b/docs/news/posts/blogger-post-2012-07-24-1.md new file mode 100644 index 0000000..f6841e5 --- /dev/null +++ b/docs/news/posts/blogger-post-2012-07-24-1.md @@ -0,0 +1,7 @@ +--- +date: 2012-07-24 +--- + +# Systemwide downtime +
+Unscheduled maintenance on virtual machine hypervisors made many OCF services unavailable between 3am and 6:30am. A somewhat-related issue affected web hosting yesterday.
diff --git a/docs/news/posts/blogger-post-2012-07-29-1.md b/docs/news/posts/blogger-post-2012-07-29-1.md new file mode 100644 index 0000000..fcde33c --- /dev/null +++ b/docs/news/posts/blogger-post-2012-07-29-1.md @@ -0,0 +1,19 @@ +--- +date: 2012-07-29 +--- + +# File server and user directory migration +
+We will be migrating the NFS disk array (file server) and LDAP server (user directory) to new hardware after 8pm tonight. This is necessary to maintain uptime and acceptable performance as much as possible during and after our move to Hearst Gym in August.
+
+We will attempt to keep files read-only where possible so that services including web hosting will not be as severely affected.
+
+We lack the extra hard drives on hand (we will be using the same hard drives in the new hardware, which means they must be backed up, formatted, and restored), so partial downtime of all OCF services will be unavoidable.
+
+Edit 07/29 02:00am:  LDAP migration completed successfully without service interruption. NFS migration (during which file access will be read-only) postponed to later today.
+
+Edit 07/29 06:00pm:  NFS migration started. Mail service is offline. Web hosting and SSH are read-only.
+
+Edit 07/30 01:10am:  All services except mail restored.
+
+Edit 07/30 01:25am:  All services restored. NFS disk array may require minor downtime in the near future. Good night.
diff --git a/docs/news/posts/blogger-post-2012-08-04-1.md b/docs/news/posts/blogger-post-2012-08-04-1.md new file mode 100644 index 0000000..24ee3cf --- /dev/null +++ b/docs/news/posts/blogger-post-2012-08-04-1.md @@ -0,0 +1,9 @@ +--- +date: 2012-08-04 +--- + +# Mail is down +
+OCF email service abruptly stopped due to an issue in our authentication system. Please check back for updates.
+
+Problem was resolved. (2:27 pm 8/4)
diff --git a/docs/news/posts/blogger-post-2012-08-12-1.md b/docs/news/posts/blogger-post-2012-08-12-1.md new file mode 100644 index 0000000..4df328e --- /dev/null +++ b/docs/news/posts/blogger-post-2012-08-12-1.md @@ -0,0 +1,11 @@ +--- +date: 2012-08-12 +--- + +# Backup server compromised +
+We discovered that our backup server (pollution.ocf) had been recently misconfigured and was subsequently compromised. pollution maintains copies of account data (home directories, web directories, MySQL databases, and mail inboxes) and administrative credentials.
+
+We immediately investigated, changed the affected credentials, and notified campus. Please assume that the attackers had access to a copy of your account data, and take appropriate measures, especially if you store sensitive or restricted data. Although we determined that account credentials (e.g., password hashes), which are not stored on pollution, were not compromised, feel free to reset your password as an additional security measure.
+
+We sincerely apologize for this inconvenience. If you need any assistance, please let us know.
diff --git a/docs/news/posts/blogger-post-2012-08-15-1.md b/docs/news/posts/blogger-post-2012-08-15-1.md new file mode 100644 index 0000000..4ffe3c2 --- /dev/null +++ b/docs/news/posts/blogger-post-2012-08-15-1.md @@ -0,0 +1,8 @@ +--- +date: 2012-08-15 +--- + +# Mail outage +
+Mail was unavailable between 9:30pm Monday and 7pm Tuesday as a result of electrical work for the UPS, which broke NFS, see post Moving to Hearst Gym. We apologize for the inconvenience.
+ diff --git a/docs/news/posts/blogger-post-2012-08-16-1.md b/docs/news/posts/blogger-post-2012-08-16-1.md new file mode 100644 index 0000000..3548593 --- /dev/null +++ b/docs/news/posts/blogger-post-2012-08-16-1.md @@ -0,0 +1,12 @@ +--- +date: 2012-08-16 +--- + +# Unexpected downtime +
+NFS home directories were unavailable between 6:30pm yesterday and 9:30am this morning. Most OCF services were affected. The outage appears to be related to networking changes for the move to Hearst Gym.
+
+Edit 4:00pm: We discovered that the mail server ran out of space during the outage and was mishandling mail between 6:25am and now. Domino effect.
+
+Edit Fri 4:00am: Unfortunately parts of the mail server became corrupted. Mail was being temporarily rejected until we resolved numerous software issues by 3:00am. Delayed mail should now be coming in.
+ diff --git a/docs/news/posts/blogger-post-2012-08-30-1.md b/docs/news/posts/blogger-post-2012-08-30-1.md new file mode 100644 index 0000000..c02dc6b --- /dev/null +++ b/docs/news/posts/blogger-post-2012-08-30-1.md @@ -0,0 +1,8 @@ +--- +date: 2012-08-30 +--- + +# Unexpected interruption of services +Due to issues with how NFS clients connect to the NFS server, machines were rebooted to fix the problem, including sandstorm (mail server), death (web server), and tsunami (login server). This issue appears to have started around 10:00AM. We apologize for the inconvenience.
+
+All issues should have been resolved as of 2:20PM. diff --git a/docs/news/posts/blogger-post-2012-09-04-1.md b/docs/news/posts/blogger-post-2012-09-04-1.md new file mode 100644 index 0000000..b6d63d6 --- /dev/null +++ b/docs/news/posts/blogger-post-2012-09-04-1.md @@ -0,0 +1,418 @@ +--- +date: 2012-09-04 +--- + +# Moving to Hearst Gym +
+ + + +
Hearst Gym map
PDF version
+
+
+
+
+
+
+
+
+
+
+
+

+ + + + + + + + + + + + + + + + + + + + +June 21 report

+
+As part of the Bears Initiative and Lower Sproul Redevelopment, the OCF, like many other student groups in Eshleman, will be moved to a temporary location in Hearst Gym ("the Grotto") around August 2012. We hope to see you during the fall semester in Hearst Gym.
+
+We will do our best to ensure that the move proceeds smoothly, but please bear with us as we cope with the well-intentioned powers that be. There may be server downtime sometime between July and December 2012 (to be announced).
+
+The OCF will continue to offer its services moving forward, including a computer lab, although the lab space in Hearst Gym will be smaller.
+
+

+ + + + + + + + + + + + + + + + + + + + +July 8 report

+
+
+

+ + + + + + + + + + + + + + + + + + + + +

+

+ + + + + + + + + + + + + + + + + + + + +Lab

+
+We're planning to reopen the lab in Hearst Gym on the first day of fall semester, Thursday, August 16th Tuesday, August 21st Monday, August 27th Tuesday, September 4th (see below) with the same hours as before (dependent on the new building hours which we're working to address resolved, see below). Our tentative move date is Saturday, August 11th, and we don't have access to the space before [or after] then, so this doesn't give us a lot of time.
+
+
+
+The new lab space is significantly smaller. Currently we have 28 desktop computers; in Hearst Gym we'll have 16 desktop computers and 2 printers. Considering that members (including OCF staff!) often had to wait for an available workstation as it is, this is problematic, but it's the only possible compromise (we were planning on five more but ASUC rejected them last week because of space constraints).
+
+In terms of creature comforts, there will be a single small couch (right now we have three large ones), possibly some comfy chairs (right now we have five), and no refrigerator (right now we have a refrigerator, microwave, and toaster).
+
+ + + +
(via Donna Davis)
+
+
+We have a lot of stuff (this includes our server room 22,200 BTU interior/exterior A/C units, remaining monitors/desktops, whiteboard, chairs, and cables) that we'd like to keep but have no space for, so we're looking into our options (donating, selling, giving away, disposing). Ideas?
+
+

+ + + + + + + + + + + + + + + + + + + +

+

+ + + + + + + + + + + + + + + + + + + +Online services

+
+We're hoping to keep some infrastructure behind in Eshleman so that web hosting, email, remote shell, file access, etc. won't be down for more than a few hours on our move date. This depends on how early we will have access to Hearst Gym and how late we will have access to our existing server room, which we're still working to address.
+
+
+

+ + + + + + + + + + + + + + + + + + + + +Putting everything in perspective

+
+Two years ago OCF completed its 15-month move from Heller Lounge in MLK +to the larger "permanent resting space" in Eshleman, with excessive lab and server downtime +(caused in part by the discovery of asbestos and later a failed pump flooding the new lab, then a pipe burst dropping raw sewage, then a network and later power outage which was prolonged). Once again, we're moving, except this time our space is smaller, the timeframe is shorter, and budgets are tighter...
+
+
+
+17 months of emails, phone calls, and meetings with the ASUC, Auxiliary, Capital Projects, architects, and planners: we're frustrated. Frustrated by the iterations of plans and people that come and go, leaving us to start all over again. So frustrated that the impending move is something to look forward to.
+
+But please don't point fingers at anybody: it's easy to blame the University or the ASUC. The reality is complicated and decisions are not easy. "They have a difficult job in that they are trying to cut a cupcake into a million little pieces."
+
+Here are some kind words during our last move from our friends upstairs at the Daily Californian that mean so much to us:
+
+OCF Languishes, Reopening To Be Postponed Further
+by Danica Li, The Daily Clog (Sep 17, 2008)
+
+Everyone loves the Open Computing Facility. It’s got a public refrigerator, microwave, and toaster, clever decorative tastes, free web space, and a central, lower Sproul location. 250 pages of free printing a semester ain’t something that’s easy to turn down, either.
+
+Unceremoniously evicted from their sweet Heller Lounge digs at the end of last semester, the OCF has had to pack their things and haul ass to the basement of Eshleman Hall. The computer lab was scheduled to open by the beginning of the semester. So what’s holding things up?
+
+Apparently, construction’s stalled because the OCF are waiting on funds from the ASUC. They also discovered a leak under the flooring in the lounge of the Publication Center, so OCF volunteers — bless their souls — have been scrambling to get that fixed as well.
+
+The target reopening date is still listed as mid to late September on the website. Godspeed, guys. You provide a top-notch public service, and we wish you the best of luck in hassling ASUC for funds.
+Hopefully you'll still love us without the refrigerator, microwave, and toaster and with increasingly inconvenient locations (2nd fl MLK Heller Lounge, basement Eshleman, basement Hearst Gym).
+
+We've still got major leaks when it rains, undesirable guests (cockroaches and mice, not the computer kind, even in our spotless but comfy server room -- it's time to get a cat, see below). There are some things we won't miss. We'll miss our self-assembled tables and hardwood floors though.
+ + + +
(via lolcat.com)
+
+
+

+July 26 report

+
+
+Good news:
+
+
    +
  • Progress is being made.
    • +
  • Hearst building hours will not conflict with our current hours.
    • +
+Bad news:
+
+
    +
  • Hearst basement ("the Grotto") opens on Tuesday, August 21st so we have to delay opening the lab until then (or later, see below).
    • +
  • IS&T may delay wiring the area to the campus network (but this is not definitive), without which we don't have an Internet connection, and wouldn't be able to move our servers or run the lab.
    • +
+
+

+August 7 report

+
+
+
+
+Good news:
+
    +
  • Two rack enclosures were moved to the work-in-progress server room in Hearst Gym on Friday, August 3rd.
    • +
+
+
+Bad news:
+
+
    +
  • Floor installation (needed before installation of furniture and equipment) is behind schedule.
    • +
  • Cardkey reader installation (needed for access to space) is behind schedule.
    • +
  • Saturday, August 11th move was canceled (uncanceled, see below) by the University this evening because of code compliance issues. University is limited in staff and has commitments to other projects (e.g., the Stadium, Helios, Martinez Commons). The new move date has not yet been scheduled.
    • +
+
+
+

+August 9 report

+
+Good news:
+
    +
  • We've been given the green light to move on Saturday, August 11th at 8:30am.
    • +
+
+
+Bad news:
+
+
+
    +
  • We won't have access to Hearst Gym space, so we will have to keep most servers and some other equipment in Eshleman.
    • +
+
+
+

+August 11 report

+
+
+ + + +
Movers were very helpful.
+
+
+
+ + + +
Server room
+ + + +
Lab space.
+With large conduits and ducts.
+
+Good news:
+
    +
  • We moved some of our equipment (mostly lab computers, chairs, +boxes) this morning.
    • +
+
+Bad news:
+
+
    +
  • There's still a lot we couldn't move because furniture, data connectivity, and power connections are not ready.
    • +
+
+
+

+August 13 report

+
+UPS was disconnected today and will be moved tomorrow. Because we were only given 15-minute advance notice of the electrical work, we had to shutdown many servers ahead of time and as a result there was a short interruption in all services around 2pm and 9:30pm.
+
+

+August 14 report

+
+University aborted UPS move. Not yet rescheduled.
+
+We also discovered a mail outage caused by the electrical work last night. We apologize for the inconvenience.
+
+

+August 16 report

+
+UPS move completed. Electrical work being worked on. Still no network uplink.
+
+We discovered a second outage caused by some networking work last night. We apologize for the inconvenience.
+
+

+August 22 report

+
+We haven't published an update in a while.  A lot has changed, but there has also been plenty of frustration.
+
+
+Good news:
+
    +
  • Electrical work in server room was completed yesterday.
    • +
  • Data connectivity was completed today and we have a new working subnet: 169.229.172.64/26
    • +
  • Cardkey readers became operational today (but see below, we don't actually have cardkey access because there is some bureaucratic work involved).
    • +
+
+Bad news:
+
    +
  • We don't have access to space. University is still processing paper access forms, and although we've done our best to expedite them, there are ~600 access forms that need be processed manually alongside ours. Actually, we've been piggybacking off the contractors during their hours to access the space (but that's not feasible tomorrow once classes begin).
    • +
  • Some tables are still missing.
    • +
+
+
+Given that we don't have access, let alone time to set anything up, we're postponing lab opening to (at the earliest), Monday, August 27, 2012 Tuesday, September 4th.
+
+

+August 25 report

+
+
+
+Unfortunately, since we still don't have physical access to our own space, we are AGAIN postponing our lab opening to Tuesday, September 4th.
+
+

+August 29 report

+
+
+You know what they say about academia...our access form paperwork still keeps shifting around, and it's possible that we won't get access until next week (which means we'd have to delay opening again).
+
+

+September 2 report

+
+One +of us got access to the new lab Saturday night, so we're working "around + the clock" and laboring on Labor Day to get a partial lab up on +Tuesday.
+
+ + + +
Discreetly transporting computer supplies. At midnight.


+
+
+

+September 3 report

+
+
+
+Half of lab (7 desktops) and limited printing will be open on Tuesday, September 4th. Because the lab setup was rushed, some things may be broken, and please let the Publications Center front desker know.
+
+

+September 19 report

+
+
+
+Lab was fully opened last week, but there will still be some construction and furniture rearrangement (some of which is currently missing) in the meantime. Temporary AirBears was set up by IS&T yesterday.
+
+Servers (including lab backend) are staying in Eshleman and will be moved over winter break beginning Saturday, December 15, during which intermittent downtime (for web hosting, SSH, email, etc.) is to be expected.
+
+
+
+
+
+
    +
+
+
+
+ diff --git a/docs/news/posts/blogger-post-2012-09-07-1.md b/docs/news/posts/blogger-post-2012-09-07-1.md new file mode 100644 index 0000000..9ee4cd8 --- /dev/null +++ b/docs/news/posts/blogger-post-2012-09-07-1.md @@ -0,0 +1,12 @@ +--- +date: 2012-09-07 +--- + +# Incorrect CGI ownership on webserver +
+For approximately 8 hours, (between 7:15am and 3:30pm on Sep 5), CGI scripts were running as shared user www-data. This may have broken CGI scripts during that time and if directories were world-writable (not recommended for web directories), may have created files owned by www-data. We are investigating and will correct any bad file ownerships as needed.
+
+PHP (more commonly used than CGI) was not affected.
+
+Edit 01:00am: File ownerships were corrected. Only one user was affected.
+ diff --git a/docs/news/posts/blogger-post-2012-10-07-1.md b/docs/news/posts/blogger-post-2012-10-07-1.md new file mode 100644 index 0000000..ccda18b --- /dev/null +++ b/docs/news/posts/blogger-post-2012-10-07-1.md @@ -0,0 +1,29 @@ +--- +date: 2012-10-07 +--- + +# Election 2012 +
+
+Your political voice matters - this message brought to you by the non-partisan Open Computing Facility Political Action Committee (OCF PAC).
+
+
+

+Register to vote

+
+Register to vote online by October 22. +Am I registered? (Links are for California voters, other states may have different deadlines and registration requirements.)
+
+
+

+Prop 30: Temporary Taxes for Education

+
+Spread the awareness about CA Prop 30: Temporary Taxes for Education (non-partisan research study at UC Berkeley by the CITRIS Data and Democracy Initiative and AMP Lab).
+
+ + + +
via www.internetvotes.org
+
+
+ diff --git a/docs/news/posts/blogger-post-2012-10-28-1.md b/docs/news/posts/blogger-post-2012-10-28-1.md new file mode 100644 index 0000000..84415b9 --- /dev/null +++ b/docs/news/posts/blogger-post-2012-10-28-1.md @@ -0,0 +1,10 @@ +--- +date: 2012-10-28 +--- + +# Downtime due to updates +
+We updated our servers and need to reboot them. Expect intermittent downtime starting at 10:20pm.
+
+Update: completed successfully.
+ diff --git a/docs/news/posts/blogger-post-2012-11-10-1.md b/docs/news/posts/blogger-post-2012-11-10-1.md new file mode 100644 index 0000000..7f8169a --- /dev/null +++ b/docs/news/posts/blogger-post-2012-11-10-1.md @@ -0,0 +1,12 @@ +--- +date: 2012-11-10 +--- + +# SSH server migration +
+SSH login server (ssh.ocf, tsunami.ocf) was migrated to new server. SSH and SFTP will be unavailable for approximately an hour.
+
+The new server is running on newer and faster hardware that will be easier to migrate during the move to Hearst Gym after the semester ends.
+
+Update Nov 11: Migration was completed on Friday. Old SSH host keys were restored this morning, so you should not be receiving warnings about host keys changing since before the migration. Please contact us about any new issues that you may encounter.
+ diff --git a/docs/news/posts/blogger-post-2012-11-28-1.md b/docs/news/posts/blogger-post-2012-11-28-1.md new file mode 100644 index 0000000..ca56493 --- /dev/null +++ b/docs/news/posts/blogger-post-2012-11-28-1.md @@ -0,0 +1,12 @@ +--- +date: 2012-11-28 +--- + +# Eshleman lockdown +
+UCPD has locked down Eshleman Hall, where our servers are located, until at least Sunday, in response to the Eshleman occupation.
+
+Downtime is possible as we have abruptly lost physical access to our server and networking equipment.
+
+Update 11/28 12:30pm: SSH server was inaccessible between 12:05pm and 12:25pm.
+ diff --git a/docs/news/posts/blogger-post-2012-11-29-1.md b/docs/news/posts/blogger-post-2012-11-29-1.md new file mode 100644 index 0000000..4692bd4 --- /dev/null +++ b/docs/news/posts/blogger-post-2012-11-29-1.md @@ -0,0 +1,12 @@ +--- +date: 2012-11-29 +--- + +# Possible hard drive failure +
+At 4am our primary server (hal, which is the hypervisor for most of our servers) reported two uncorrected read errors on one of its hard drives  (in a RAID 10 setup). The presence of uncorrected errors can indicate hard drive failure because it means that the hard drive has run out of spare sectors.
+
+We have a hard drive replacement on hand, but as our servers are in Eshleman (which is under a lockdown) there may be a delay getting physical access.
+
+Update Dec 3: We have taken backups of important data stored on hal, but do not have physical access to our servers at this time.
+ diff --git a/docs/news/posts/blogger-post-2012-12-03-1.md b/docs/news/posts/blogger-post-2012-12-03-1.md new file mode 100644 index 0000000..1f63374 --- /dev/null +++ b/docs/news/posts/blogger-post-2012-12-03-1.md @@ -0,0 +1,8 @@ +--- +date: 2012-12-03 +--- + +# Extended lab hours for RRR and finals +
+Lab will be open from 9am to 9pm from Tuesday 12/4 to Thursday 12/13 (including weekends) and 9am to 12pm on Friday, 12/14, the last day of finals.

Lab will be closed after finals as we move our servers from Eshleman over winter break. Expect intermittent downtime for web hosting, SSH, email, etc. between December 14 and December 18 (and possibly later).
+ diff --git a/docs/news/posts/blogger-post-2013-01-18-1.md b/docs/news/posts/blogger-post-2013-01-18-1.md new file mode 100644 index 0000000..85f5b2c --- /dev/null +++ b/docs/news/posts/blogger-post-2013-01-18-1.md @@ -0,0 +1,6 @@ +--- +date: 2013-01-18 +--- + +# DNS downtime +DNS and IRC was unavailable between 12:56am and 03:22am due to a networking error. System performance was degraded, unexpected errors may have occurred during that time, and logins may have timed out. We apologize for the inconvenience. diff --git a/docs/news/posts/blogger-post-2013-02-17-1.md b/docs/news/posts/blogger-post-2013-02-17-1.md new file mode 100644 index 0000000..df80d03 --- /dev/null +++ b/docs/news/posts/blogger-post-2013-02-17-1.md @@ -0,0 +1,18 @@ +--- +date: 2013-02-17 +--- + +# File storage system (NFS) restructure +We will be restructuring our file storage system sometime over the next semester. Please be aware that the u1-u8 file structure will be gone. If you have any files that depend on that file structure, they will not work anymore. They can be fixed by changing them to use your home directory (e.g. /home/f/fo/foo instead of /mnt/u3/home/f/fo/foo).
+
+Update 1/25: We will be taking the OCF down at around 10pm (we decided to delay until 1:15am).
+
+Update 1/26: Due to routine maintenance before the restructuring, NFS will be down for at least all of Saturday. Total downtime was between 1:15am and 3:00pm.
+
+Update 1/28: NFS file storage maintenance partially completed. We will be taking the OCF down again sometime in the near-ish future to finish the maintenance.
+
+Update 2/11: We will be taking the OCF down on Saturday, February 16 to finish maintenance of the NFS file storage system.
+
+Update 2/13: Maintenance will start on Saturday after 6pm when the lab is closed. Web hosting, email, and SSH/SFTP will be intermittently inaccessible during maintenance.
+
+Update 2/17: The file storage maintenance has been completed successfully. Downtime occurred during the 11:30pm to 12:00am (30 minutes) maintenance window. Note that hard-coded references to /mnt will no longer work. diff --git a/docs/news/posts/blogger-post-2013-02-17-2.md b/docs/news/posts/blogger-post-2013-02-17-2.md new file mode 100644 index 0000000..1968fc7 --- /dev/null +++ b/docs/news/posts/blogger-post-2013-02-17-2.md @@ -0,0 +1,18 @@ +--- +date: 2013-02-17 +--- + +# Printing daily quota +Starting today, only 30 pages may be printed per day at the OCF lab by each member. Each member is still also limited to printing 250 pages per semester.
+
+

+Why are we doing this?

+ +See the February 12 OCF Board of Directors meeting minutes. + diff --git a/docs/news/posts/blogger-post-2013-04-12-1.md b/docs/news/posts/blogger-post-2013-04-12-1.md new file mode 100644 index 0000000..7e96c45 --- /dev/null +++ b/docs/news/posts/blogger-post-2013-04-12-1.md @@ -0,0 +1,6 @@ +--- +date: 2013-04-12 +--- + +# Campus-wide outage +The entire campus appears to have been inaccessible from outside between 12:19am and 2:11am, so OCF was inaccessible from off-campus as well. We do not yet know the cause of the widespread outage. diff --git a/docs/news/posts/blogger-post-2013-04-28-1.md b/docs/news/posts/blogger-post-2013-04-28-1.md new file mode 100644 index 0000000..f7e2099 --- /dev/null +++ b/docs/news/posts/blogger-post-2013-04-28-1.md @@ -0,0 +1,6 @@ +--- +date: 2013-04-28 +--- + +# Email virtual hosting downtime +Email virtual hosting (e.g., email forwarding for person@group.berkeley.edu) was unavailable today, April 27, between 2am and 11:30pm. Email senders during this time would have received a bounce message that said "Relay access denied". We apologize for the inconvenience. diff --git a/docs/news/posts/blogger-post-2013-06-30-1.md b/docs/news/posts/blogger-post-2013-06-30-1.md new file mode 100644 index 0000000..9f6d8fd --- /dev/null +++ b/docs/news/posts/blogger-post-2013-06-30-1.md @@ -0,0 +1,10 @@ +--- +date: 2013-06-30 +--- + +# Unexpected downtime as services are inaccessible +The OCF webserver (www.ocf/death.ocf) and login servers (ssh.ocf/tsunami.ocf,supernova.ocf) are offline.  They are virtual machines hosted on hal.ocf which is still running and responds to network requests.
+
+No current estimate on when the servers will be back up.  I am contacting site managers now.
+
+Update 1am:  Network problems fixed. All services back up. diff --git a/docs/news/posts/blogger-post-2013-07-15-1.md b/docs/news/posts/blogger-post-2013-07-15-1.md new file mode 100644 index 0000000..68b1bef --- /dev/null +++ b/docs/news/posts/blogger-post-2013-07-15-1.md @@ -0,0 +1,12 @@ +--- +date: 2013-07-15 +--- + +# ocf.berkeley.edu now points to web server +After nearly three years of deprecating the ocf.berkeley.edu hostname for SSH/SFTP access to shell accounts,  we have decided to point ocf.berkeley.edu to the web server.
+
+The good news is that ocf.berkeley.edu in web browsers will seamlessly redirect (302 permanent redirect) to www.ocf.berkeley.edu. Less keystrokes, less to remember.
+
+The bad news is that you cannot access your shell account using the hostname ocf.berkeley.edu. You will need to use ssh.ocf.berkeley.edu as has been listed in our wiki documentation.
+
+We believe that the compromise is worth it especially in light of the long transition period. diff --git a/docs/news/posts/blogger-post-2013-08-12-1.md b/docs/news/posts/blogger-post-2013-08-12-1.md new file mode 100644 index 0000000..b764021 --- /dev/null +++ b/docs/news/posts/blogger-post-2013-08-12-1.md @@ -0,0 +1,20 @@ +--- +date: 2013-08-12 +--- + +# Shorter URLs for web hosting +In our last blog post, we announced that ocf.berkeley.edu points to the web server, meaning that you can now leave off the "www" in www.ocf.berkeley.edu.
+
+We're now announcing ocf.io as a shorter alternative to ocf.berkeley.edu, "in beta".
+
+In addition, if you leave off the tilde ("~") for an account name, it will be added.
+
+In other words, the following are shorthand forms of the canonical form http://www.ocf.berkeley.edu/~account :
+ +The shorthand URL forms are 302 redirected ("moved permanently") to the canonical URL. diff --git a/docs/news/posts/blogger-post-2013-09-03-1.md b/docs/news/posts/blogger-post-2013-09-03-1.md new file mode 100644 index 0000000..ab40d63 --- /dev/null +++ b/docs/news/posts/blogger-post-2013-09-03-1.md @@ -0,0 +1,13 @@ +--- +date: 2013-09-03 +--- + +# Shell rc file changes +Due to upgrades of our servers to better supported systems, we will not be supporting old rc file locations. Please update your shell rc files accordingly. Please contact us if you have any questions or concerns.
+
+The list below shows where the new files are located. (e.g. /folder/a --> /folder/b means /folder/a is now located at /folder/b)
+
+/opt/local/environment --> /opt/ocf/share/environment
+/usr/local/environment --> /opt/ocf/share/environment
+/opt/ocf/environment --> /opt/ocf/share/environment
+/usr/ucb/whoami --> /usr/bin/whoami diff --git a/docs/news/posts/blogger-post-2013-09-03-2.md b/docs/news/posts/blogger-post-2013-09-03-2.md new file mode 100644 index 0000000..eefa7dd --- /dev/null +++ b/docs/news/posts/blogger-post-2013-09-03-2.md @@ -0,0 +1,12 @@ +--- +date: 2013-09-03 +--- + +# New Debian Wheezy ssh.ocf (tsunami) server +On Monday, September 2, the existing ssh.ocf server will be upgraded from Debian Squeeze to Debian Wheezy. More updates will be posted as that date approaches.
+
+EDIT 9/2/13 2:04pm: upgrade happening now
+
+EDIT 4:17pm: progressing along, should be done within the next hour
+
+EDIT 5:02pm: the upgrade has been completed. Please contact us if there are any issues. diff --git a/docs/news/posts/blogger-post-2013-10-01-1.md b/docs/news/posts/blogger-post-2013-10-01-1.md new file mode 100644 index 0000000..698a92c --- /dev/null +++ b/docs/news/posts/blogger-post-2013-10-01-1.md @@ -0,0 +1,9 @@ +--- +date: 2013-10-01 +--- + +# Unexpected downtime - campus-wide power failure +
+OCF is experiencing unexpected downtime due to a widespread power failure on the UC Berkeley campus.  We expect power to be restored by tomorrow (Oct 1st) morning and our services should be back up by noon tomorrow.
+
+Update Oct 1 9:47am: All services back up. diff --git a/docs/news/posts/blogger-post-2013-10-07-1.md b/docs/news/posts/blogger-post-2013-10-07-1.md new file mode 100644 index 0000000..4971d3c --- /dev/null +++ b/docs/news/posts/blogger-post-2013-10-07-1.md @@ -0,0 +1,8 @@ +--- +date: 2013-10-07 +--- + +# SSH server (tsunami) maintenance +The ssh server will be restarted for maintenance over the next hour.
+
+Update 6:30am: This work has been completed successfully. diff --git a/docs/news/posts/blogger-post-2013-10-07-2.md b/docs/news/posts/blogger-post-2013-10-07-2.md new file mode 100644 index 0000000..79586ea --- /dev/null +++ b/docs/news/posts/blogger-post-2013-10-07-2.md @@ -0,0 +1,27 @@ +--- +date: 2013-10-07 +--- + +# Printing service changes +
+As an organization managed and operated entirely by student volunteers, we don't have the financial resources, equipment, or facility space to sustain our current and projected levels of printing demand.
+
+We have already implemented a number of measures to manage our levels of printing, including:
+ +Despite measures, we estimate that we will print over 500,000 pages this semester and thus exceed our planned printing budget. While they have been largely successful, the savings have been offset by increases in +membership without an increase in funding or student volunteers.
+
+In addition to the budgetary challenges, our printing equipment is not designed for our current levels of +printing and require constant repair due to frequent toner leaks, which +reduce reliability and print quality.
+
+In an effort to reduce costs and increase reliability, the OCF Board of Directors has introduced a weekday printing limit of 10 pages/day. The existing 30 page/day limit will currently continue on weekends. OCF may make additional changes to printing policy this semester and future semesters to manage increasing printing demand.
+
+We recognize that this will greatly inconvenience students who do not own or otherwise have access to a printer, and we hope that this will encourage the university to consider affordable alternatives. We welcome feedback in person during our weekly meetings on Thursdays at 8pm in the California Conference Room (across the lab).
+
+See the October 3 Board of Directors meeting minutes details on the decision and the alternatives that we considered.
+ diff --git a/docs/news/posts/blogger-post-2013-10-11-1.md b/docs/news/posts/blogger-post-2013-10-11-1.md new file mode 100644 index 0000000..fe263e6 --- /dev/null +++ b/docs/news/posts/blogger-post-2013-10-11-1.md @@ -0,0 +1,10 @@ +--- +date: 2013-10-11 +--- + +# MySQL database maintenance +From 1:00am to 3:00am, there will be maintenance on the MySQL database. More info will be provided along the way.
+
+Update 3:30am: maintenance is still progressing after taking longer than expected during the initial backup phase. more updates later.
+
+Update 4:20am: maintenance has been completed successfully. diff --git a/docs/news/posts/blogger-post-2013-11-10-1.md b/docs/news/posts/blogger-post-2013-11-10-1.md new file mode 100644 index 0000000..a05ba04 --- /dev/null +++ b/docs/news/posts/blogger-post-2013-11-10-1.md @@ -0,0 +1,16 @@ +--- +date: 2013-11-10 +--- + +# Email (sandstorm), web (death), and printhost server maintenance tonight +There will be maintenance on the email, webserver, and printhost servers tonight. More info will be provided along the way.
+
+Update 10:10pm: email server is under maintenance now.
+
+Update 10:40pm: email maintenance has finished successfully.
+
+Update 11:05pm: web server under maintenance now.
+
+Update 11:36pm: web server maintenance has finished successfully.
+
+Update 11:55pm: printhost (as well as the printers) are having issues. diff --git a/docs/news/posts/blogger-post-2013-11-10-2.md b/docs/news/posts/blogger-post-2013-11-10-2.md new file mode 100644 index 0000000..971d4f9 --- /dev/null +++ b/docs/news/posts/blogger-post-2013-11-10-2.md @@ -0,0 +1,18 @@ +--- +date: 2013-11-10 +--- + +# Richard Stallman: A Free Digital Society +Join UC Berkeley students, faculty and staff in Soda Hall for a lecture on A Free Digital Society by the software developer and software freedom activist Richard Stallman.
+
+Dr. Stallman founded the GNU Project and the Free Software Foundation. He is the original author of Emacs, GCC, and the General Public License. GNU software, with a Linux kernel, is used on millions of computers today.
+
+OPEN TO THE PUBLIC
+
+Date: Saturday, November 16, 2013
+Time: 5:00 p.m. Attendance is first-come, first-served.
+Venue: 306 Soda (HP Auditorium) 100 Genetics and Plant Biology Building
+
+More details: ocf.io/rms
+
+Update: Venue has changed to 100 GPB due to overwhelming demand. diff --git a/docs/news/posts/blogger-post-2013-11-21-1.md b/docs/news/posts/blogger-post-2013-11-21-1.md new file mode 100644 index 0000000..c51cce6 --- /dev/null +++ b/docs/news/posts/blogger-post-2013-11-21-1.md @@ -0,0 +1,48 @@ +--- +date: 2013-11-21 +--- + +# Email service changes +
+In an effort to increase the quality of our offered services, the Board of Directors is phasing out email services. Over the next few months, a two-phase plan will give our members time to retrieve their data and transition to alternatives.
+
+Note: Email virtual hosts for groups are not affected by Phase 2.
+
+

+Phase 1: Forward-only

+
+Effective Sunday, October 20
+
+OCF currently supports a few ways to read mail received by username@ocf.berkeley.edu:
+
    +
  1. Forwarding to another email address
    2. +
  2. IMAP, POP (e.g., Thunderbird, Apple Mail)
    3. +
  3. Web-based email interface (webmail)
    4. +
  4. Command-line clients (e.g., mutt)
    5. +
+On October 20, OCF email service will become forward-only. OCF will no longer support the storage and access of mail on OCF servers (2-4).
+
+If you do not already use forwarding, create a .forward file in your home directory (~/.forward) containing the email address to forward to. Otherwise an attempt will be made to use the email address from CalNet (if publicly listed) or provided when the account was originally requested.
+
 +After this date, email that is stored on OCF servers (including files in /var/mail, ~/Mail, ~/mail) will be deleted. If you need assistance migrating your stored email to another service, please visit staff hours.
+
+Note: No email was deleted.
+
+
+

+Phase 2: Email service discontinued

+
+Effective date TBA
+
+At a future date (yet to be determined), email service will become internal. We will cease to support email sent from a username@ocf.berkeley.edu email address to an address from another domain (e.g., recepient@example.com). Similarly, we will reject mail from other domains sent to recepient@ocf.berkeley.edu.
+
+There are two relevant exceptions to this policy:
+ +
+We arrived at this decision after much debate and deliberation. We found that the vast majority of current students forward their OCF email and do not send emails from this address. In addition, we face a significant spam email problem requiring constant policing of email sent and received on our mail server. Accordingly, we decided to focus our volunteer efforts on different services.
+
+We welcome feedback in person during our weekly meetings on Thursdays at 8pm in the California Conference Room (across the lab).
+ diff --git a/docs/news/posts/blogger-post-2014-03-08-1.md b/docs/news/posts/blogger-post-2014-03-08-1.md new file mode 100644 index 0000000..a1dbd14 --- /dev/null +++ b/docs/news/posts/blogger-post-2014-03-08-1.md @@ -0,0 +1,8 @@ +--- +date: 2014-03-08 +--- + +# Kernel upgrades and file server maintenance +All OCF servers will be restarted late Friday night (03/07) to apply kernel security updates. File server maintenance will take place early Saturday morning, and some services may experience extended downtime. All services should be restored by 3am Saturday.
+
+Update: All servers have been restarted. File server maintenance has been postponed to late Saturday night/early Sunday morning. Downtime is expected some time between about 11pm-3am. diff --git a/docs/news/posts/blogger-post-2014-03-15-1.md b/docs/news/posts/blogger-post-2014-03-15-1.md new file mode 100644 index 0000000..07dfe0c --- /dev/null +++ b/docs/news/posts/blogger-post-2014-03-15-1.md @@ -0,0 +1,6 @@ +--- +date: 2014-03-15 +--- + +# File server maintenance +File server maintenance will take place Friday and Saturday night between about 10pm and 3am. Most OCF services will be affected. diff --git a/docs/news/posts/blogger-post-2014-03-18-1.md b/docs/news/posts/blogger-post-2014-03-18-1.md new file mode 100644 index 0000000..7f149a8 --- /dev/null +++ b/docs/news/posts/blogger-post-2014-03-18-1.md @@ -0,0 +1,8 @@ +--- +date: 2014-03-18 +--- + +# Network inaccessible +As of 7:59pm, OCF has lost connectivity to the campus network. OCF services will not be accessible. We are investigating the cause.
+
+Update 8:27pm:  Connectivity restored. The cause is not known but appears to be a temporarily disconnected link or switch on the campus end. diff --git a/docs/news/posts/blogger-post-2014-03-19-1.md b/docs/news/posts/blogger-post-2014-03-19-1.md new file mode 100644 index 0000000..8c59d87 --- /dev/null +++ b/docs/news/posts/blogger-post-2014-03-19-1.md @@ -0,0 +1,10 @@ +--- +date: 2014-03-19 +--- + +# Network issues +OCF has been experiencing network connectivity issues intermittently since Monday night. The problems appear to be due to a faulty switch generating noise on the campus end. We have been moved to a different switch temporarily while they replace the bad one. We should be moved back later today.
+
+As of 10:18am, everything is operating as normal, and our access to the outside network seems to be fully functioning.
+
+ diff --git a/docs/news/posts/blogger-post-2014-04-08-1.md b/docs/news/posts/blogger-post-2014-04-08-1.md new file mode 100644 index 0000000..a26b823 --- /dev/null +++ b/docs/news/posts/blogger-post-2014-04-08-1.md @@ -0,0 +1,6 @@ +--- +date: 2014-04-08 +--- + +# System reboot scheduled for Tuesday night +Out of an abundance of caution, we will be rebooting all OCF systems Tuesday night after 11pm in response to CVE 2014-0160. Expected downtime is less than 15 minutes. diff --git a/docs/news/posts/blogger-post-2014-04-10-1.md b/docs/news/posts/blogger-post-2014-04-10-1.md new file mode 100644 index 0000000..a8f7a8f --- /dev/null +++ b/docs/news/posts/blogger-post-2014-04-10-1.md @@ -0,0 +1,6 @@ +--- +date: 2014-04-10 +--- + +# Daily printing limit raised +The weekday printing limit has been raised from 10 pages/day to 15 pages/day for the remainder of the semester. diff --git a/docs/news/posts/blogger-post-2014-04-10-2.md b/docs/news/posts/blogger-post-2014-04-10-2.md new file mode 100644 index 0000000..b03162e --- /dev/null +++ b/docs/news/posts/blogger-post-2014-04-10-2.md @@ -0,0 +1,18 @@ +--- +date: 2014-04-10 +--- + +# CVE-2014-0160 ("Heartbleed") openssl vulnerability update +On Monday, an extremely serious bug in openssl was announced. The bug affected all recent versions of openssl, including the version in use on all OCF servers.
+
+
+Every OCF server was patched against the bug within an hour of Debian releasing a fix. However, because the bug was present in openssl for multiple years, there is no guarantee that private keys or other sensitive information (like user passwords) were not leaked.
+
+
+
+As a precaution, we have revoked all SSL certificates in use by OCF (~9 of them), discarded old private keys, and installed new certs and keys in their place.
+
+
+
+Some estimates suggest that over 66% of all HTTPS websites were vulnerable to this bug, and it is known that many high-profile sites were. We encourage you to update your OCF password, as well as your passwords on every website you use.
+ diff --git a/docs/news/posts/blogger-post-2014-04-27-1.md b/docs/news/posts/blogger-post-2014-04-27-1.md new file mode 100644 index 0000000..2cbde38 --- /dev/null +++ b/docs/news/posts/blogger-post-2014-04-27-1.md @@ -0,0 +1,6 @@ +--- +date: 2014-04-27 +--- + +# Scheduled restart Tuesday for kernel updates +All OCF servers will be restarted late Tuesday night (04/29) to apply kernel security updates. diff --git a/docs/news/posts/blogger-post-2014-06-01-1.md b/docs/news/posts/blogger-post-2014-06-01-1.md new file mode 100644 index 0000000..2742fdb --- /dev/null +++ b/docs/news/posts/blogger-post-2014-06-01-1.md @@ -0,0 +1,20 @@ +--- +date: 2014-06-01 +--- + +# Email discontinuation update: forward-only on June 14th +Back in November of last year, the OCF Board of Directors announced that email service was being discontinued, with the first phase being transitioning into forward-only email service.
+
+The start date for the first phase, October 20th, has long passed, although we never actually enforced the change. We're now ready to move into forward-only mode, and will enter this phase on Saturday, June 14th.
+
+After this date, email will be forwarded to your registered email address. For most members, this is the email you provided when you created your account, but for old accounts, or for accounts with complicated .forward or .procmailrc files, this entry may be blank or incorrect.
+
+You are strongly advised to run update-email over SSH in order to verify that you have registered an email with LDAP. If you don't have an SSH client handy, the easiest way to access SSH is by visiting ssh.ocf.berkeley.edu in your web browser.
+
+Email will only be forwarded to your email in LDAP. .forward and .procmailrc files will no longer be processed. If no email has been set after June 14th, incoming email will bounce.
+
+If you need assistance updating your email or accessing your OCF account, please get in touch so we can help you sort it out.
+
+Happy summer!
+
+Edit for clarification: Email addresses set in LDAP via update-email will only be used for forwarding after June 14th. There's unfortunately no good way to run the old system (.forward/.procmailrc files) alongside the new system, so setting an email will have no effect until the switch. Sorry for the confusion! diff --git a/docs/news/posts/blogger-post-2014-06-04-1.md b/docs/news/posts/blogger-post-2014-06-04-1.md new file mode 100644 index 0000000..6eeafb5 --- /dev/null +++ b/docs/news/posts/blogger-post-2014-06-04-1.md @@ -0,0 +1,17 @@ +--- +date: 2014-06-04 +--- + +# Subnet move +We'll be moving from our trusty old subnet, 169.229.172.64/26, to a brand new subnet 4x the size: 169.229.10.0/24. All hosts should be fully migrated by Tuesday, June 10th.
+
+
+
+There may be small periods of downtime for all servers during Monday evening/Tuesday morning, but we don't anticipate extended downtime. All services should be otherwise unaffected.
+
+
+
+If you access OCF servers via SSH, you may notice a warning that the IP address for ssh.ocf.berkeley.edu has changed. The host key will not change, so you can continue to verify against our SSH fingerprints (available via HTTPS on our wiki). The new IP address for the public login server (tsunami) will be 169.229.10.25.
+
+Update 2014/06/06: All OCF servers are now assigned both their old and new IP addresses. Old IPs have been removed from DNS, and servers will soon be assigned only their new IP. This may break existing connections (such as SSH).
+ diff --git a/docs/news/posts/blogger-post-2014-06-07-1.md b/docs/news/posts/blogger-post-2014-06-07-1.md new file mode 100644 index 0000000..ee3baf6 --- /dev/null +++ b/docs/news/posts/blogger-post-2014-06-07-1.md @@ -0,0 +1,6 @@ +--- +date: 2014-06-07 +--- + +# Kernel updates 06/07 +All OCF servers will be restarted Saturday night (06/07) to apply security updates. diff --git a/docs/news/posts/blogger-post-2014-06-17-1.md b/docs/news/posts/blogger-post-2014-06-17-1.md new file mode 100644 index 0000000..db5a984 --- /dev/null +++ b/docs/news/posts/blogger-post-2014-06-17-1.md @@ -0,0 +1,18 @@ +--- +date: 2014-06-17 +--- + +# POP/IMAP, webmail phased out +As part of our phasing out of email service for individual OCF members, we have migrated to forward-only email service. This means that no new mail can be stored, so POP/IMAP email access is no longer very useful.
+
+
+On June 21st, roughly a week from today, we will turn off POP/IMAP email access permanently for individual accounts. Webmail will also be unavailable after this date. We are leaving this window as we understand some users may wish to archive their emails by downloading them via POP/IMAP.
+
+
+
+After this date, your mailbox will be moved into your home directory where you can access it at any time (we will never delete any mail). Your mailbox is in standard mbox format, which can be opened by almost any email client, or even a simple text editor.
+
+
+
+If you have any trouble downloading your mail or accessing your OCF account, don't hesitate to reach out to us!
+ diff --git a/docs/news/posts/blogger-post-2014-07-15-1.md b/docs/news/posts/blogger-post-2014-07-15-1.md new file mode 100644 index 0000000..7911b73 --- /dev/null +++ b/docs/news/posts/blogger-post-2014-07-15-1.md @@ -0,0 +1,6 @@ +--- +date: 2014-07-15 +--- + +# Kernel updates 07/18 +All OCF servers will be restarted Friday night (07/18) in order to apply security updates. We will also be increasing the memory and CPU allocated to tsunami (the login server) during this time. diff --git a/docs/news/posts/blogger-post-2014-08-01-1.md b/docs/news/posts/blogger-post-2014-08-01-1.md new file mode 100644 index 0000000..dff73f5 --- /dev/null +++ b/docs/news/posts/blogger-post-2014-08-01-1.md @@ -0,0 +1,10 @@ +--- +date: 2014-08-01 +--- + +# Virtual hosting email downtime +Virtual hosting email service was unavailable during the last two weeks. Misconfiguration with our spam filter was unfortunately overlooked over summer. Thanks to Berkeley Consulting for bringing this to our attention.
+
+Email messages sent in the last 3-5 days might still be delivered. Older messages would have "bounced" to the sender.
+
+ We apologize for this inconvenience. diff --git a/docs/news/posts/blogger-post-2014-08-05-1.md b/docs/news/posts/blogger-post-2014-08-05-1.md new file mode 100644 index 0000000..c0a9aa4 --- /dev/null +++ b/docs/news/posts/blogger-post-2014-08-05-1.md @@ -0,0 +1,8 @@ +--- +date: 2014-08-05 +--- + +# Campus-wide network issues +As of about 9:00am Tuesday morning, Berkeley campus has been experiencing severe network issues, including high packet loss and latency. Open Computing Facility servers are affected by this outage.
+
+This is a problem on IST's side; updates are available from them. diff --git a/docs/news/posts/blogger-post-2014-08-11-1.md b/docs/news/posts/blogger-post-2014-08-11-1.md new file mode 100644 index 0000000..a9c5896 --- /dev/null +++ b/docs/news/posts/blogger-post-2014-08-11-1.md @@ -0,0 +1,10 @@ +--- +date: 2014-08-11 +--- + +# Online account tools maintenance +OCF's online account tools will be unavailable for a few days while we perform maintenance and upgrades. During this time, requesting an account and resetting passwords via CalNet will not be possible.
+
+There will be no impact on LDAP or other services, and password changes (assuming you know the old password) can be done via the "passwd" command.
+
+Update 2014-08-18: We've completed the maintenance on our online account tools. Note that the URLs have changed; see the wiki or main website for updated links. diff --git a/docs/news/posts/blogger-post-2014-08-11-2.md b/docs/news/posts/blogger-post-2014-08-11-2.md new file mode 100644 index 0000000..1586f1b --- /dev/null +++ b/docs/news/posts/blogger-post-2014-08-11-2.md @@ -0,0 +1,10 @@ +--- +date: 2014-08-11 +--- + +# WordPress xmlrpc exploit +WordPress recently announced a bug in their xmlrpc implementation which can result in denial of service attacks by using large amounts of CPU. Many OCF-hosted sites are running affected versions of WordPress.
+
+In response to activity which took out our webserver for about 15 minutes earlier tonight, we are blocking access to xmlrpc.php files to protect the shared OCF webserver. If you would like to request xmlrpc.php files be unblocked from your site, please contact us.
+
+Now would be a good opportunity to make sure all the software on your website is up-to-date! diff --git a/docs/news/posts/blogger-post-2014-10-10-1.md b/docs/news/posts/blogger-post-2014-10-10-1.md new file mode 100644 index 0000000..b3e2212 --- /dev/null +++ b/docs/news/posts/blogger-post-2014-10-10-1.md @@ -0,0 +1,10 @@ +--- +date: 2014-10-10 +--- + +# mirrors.ocf.berkeley.edu - outdated Debian package repos +Today we noticed that our Debian package mirrors had not receive updates since October 2nd, although syncs had been completing successfully. This was due to an issue with mirrors.kernel.org, our upstream mirror, which had not synced in the past week.
+
+Since our syncs were completing normally, we weren't alerted to the problem until today when apt warned us that our mirror was out-of-date. We didn't really consider the possibility that mirrors.kernel.org, which is a top-tier Debian mirror (and one of several which ftp.us.debian.org can resolve to) would receive no updates for an extended period of time. We'll add additional health checks to make sure that not only are syncs completing, but that we're receiving updates.
+
+We contacted the mirrors.kernel.org admins to report the problem, who were very responsive and fixed the problem within 30 minutes. Our mirror is now up-to-date. diff --git a/docs/news/posts/blogger-post-2014-10-24-1.md b/docs/news/posts/blogger-post-2014-10-24-1.md new file mode 100644 index 0000000..8212938 --- /dev/null +++ b/docs/news/posts/blogger-post-2014-10-24-1.md @@ -0,0 +1,28 @@ +--- +date: 2014-10-24 +--- + +# Moving www.ocf.berkeley.edu to HTTPS-only +On November 22nd, all websites hosted under the www.ocf.berkeley.edu domain will begin using HTTPS instead of plain HTTP. We will redirect all requests to the corresponding HTTPS site.
+
+Virtually-hosted websites will continue to use plain HTTP, so won't be affected. There also shouldn't be any impact on sites which consist of static content.
+
+We recommend to change all of your absolute links to point to the HTTPS version. You can start doing this immediately; it's already supported! Since we will set up a 301 redirect, no links will be broken.
+
+There are a few things which some sites are currently doing which will cause problems under HTTPS:
+
+ +
+If your website does either of the two thing above, you must fix it before November 22nd, or parts of your website may break. If you use WordPress or another CMS, it is usually sufficient to update the URL in the admin panel.
+
+
+
+We appreciate that this may be disruptive to many websites, and have not made the decision lightly. There are many reasons to use HTTPS by default, even for websites which don't contain sensitive information or collect passwords. This is an inevitable transition for the OCF at some point, and while it will be painful, it is easier to make it now than in another four years, when even more sites will be affected.
+
+
+
+We estimate that there are about 90 OCF users (~0.3% of OCF users) with active websites which will be affected by the change. We'll try to contact them individually about the change. If you need assistance making changes to your website, don't hesitate to come in during staff hours or send us an email.
+ diff --git a/docs/news/posts/blogger-post-2014-10-26-1.md b/docs/news/posts/blogger-post-2014-10-26-1.md new file mode 100644 index 0000000..36e0019 --- /dev/null +++ b/docs/news/posts/blogger-post-2014-10-26-1.md @@ -0,0 +1,12 @@ +--- +date: 2014-10-26 +--- + +# Announcing dedicated hosting for web applications! +We're excited to announce a new OCF service for student groups: dedicated hosting for web applications like Rails, Django, Flask, and Node.js!
+
+Previously, hosting for modern web apps was only available via FastCGI, which was difficult to set up and manage. The new service makes it possible to host any app that can bind to a socket, enabling you to run basically any type of application.
+
+OCF hosting for web apps is a pretty cool choice for student groups compared to hosting off site; it's easy to get a berkeley.edu domain name and to get support from friendly volunteer staffers during staff hours, and now it's easy to set up your app on our powerful servers, all hosted on-campus.
+
+We're opening up the new application hosting on a trial basis, and hoping to work closely with a small number of student group early adopters to work out kinks and improve the service. If you're interested, take a look at the documentation and get in touch! diff --git a/docs/news/posts/blogger-post-2014-11-04-1.md b/docs/news/posts/blogger-post-2014-11-04-1.md new file mode 100644 index 0000000..d6d9a6e --- /dev/null +++ b/docs/news/posts/blogger-post-2014-11-04-1.md @@ -0,0 +1,8 @@ +--- +date: 2014-11-04 +--- + +# Kernel updates Nov. 08 +
+All OCF servers will be restarted Saturday night (11/08) in order to apply security updates. Downtime should be no more than 15 minutes.
+ diff --git a/docs/news/posts/blogger-post-2014-12-01-1.md b/docs/news/posts/blogger-post-2014-12-01-1.md new file mode 100644 index 0000000..cf18111 --- /dev/null +++ b/docs/news/posts/blogger-post-2014-12-01-1.md @@ -0,0 +1,13 @@ +--- +date: 2014-12-01 +--- + +# WordPress XSS vulnerability; please update! +A vulnerability was recently discovered in WordPress which affects a large number of OCF web hosting users. The vulnerability can potentially allow a malicious person to hijack your session and compromise your website.
+
+All users should update immediately to the latest version of WordPress. Versions 3.9.3, 3.8.5, 3.7.5, 4.0.0, 4.0.1 are unaffected by this vulnerability, but we highly advise to always use the latest version.
+
Updating WordPress is extremely easy; it's just a single click after logging in to the admin panel.
+
+Recent versions of WordPress come with automatic updates enabled for minor releases, which can help to protect you from future vulnerabilities. We strongly recommend not disabling this feature!
+
+If we've contacted you and you need help updating your site, please don't hesitate to get in touch so that we can help! diff --git a/docs/news/posts/blogger-post-2014-12-10-1.md b/docs/news/posts/blogger-post-2014-12-10-1.md new file mode 100644 index 0000000..551812b --- /dev/null +++ b/docs/news/posts/blogger-post-2014-12-10-1.md @@ -0,0 +1,29 @@ +--- +date: 2014-12-10 +--- + +# Scheduled downtime: Dec 18, Dec 27-28, Jan 3-4 +We found out yesterday that, due to construction, Hearst Gym will have no power on Dec 27-28 and Jan 3-4. Unfortunately, all OCF services will be affected by the power outage.
+
+We're looking into ways to reduce the impact, but currently you should expect the following impacts:
+
+ +
+These services will be completely unavailable:
+
+ +
+We're working now to try to minimize the impact of the outage, and will post updates here. Please email us if you have any questions.
+
+Update 12/10: We are scheduling downtime during the evening of Thursday, December 18th to test our ability to start all servers and services remotely. Total downtime should be less than 30 minutes.
+
+Update 12/18: Maintenance for tonight is completed. Total downtime was about 45 minutes (instead of the expected 30) due to a problem with a switch after we restored power. The good news is that we caught it now rather than in a week when nobody will be around to fix it. Everything else worked as expected.
+
+ diff --git a/docs/news/posts/blogger-post-2014-12-26-1.md b/docs/news/posts/blogger-post-2014-12-26-1.md new file mode 100644 index 0000000..1290d47 --- /dev/null +++ b/docs/news/posts/blogger-post-2014-12-26-1.md @@ -0,0 +1,31 @@ +--- +date: 2014-12-26 +--- + +# Update on scheduled downtime Dec 27-28 and Jan 3-4 +Update Jan. 04: The outage is over; all services have been restored.
+
 +Update Jan. 02: We've just migrated most services to the offsite server, and taken the others offline for the second (and last) scheduled outage. We expect to be back online for good Sunday evening.
+
 +Update Dec. 28: Power was restored at 7pm PDT as expected, and all services are now back online. Everything we had planned (powering on the servers remotely via IPMI, copying files/db from the offsite host, etc.) worked great during both the transition to and away from the offsite server. We will do the same thing next weekend. If you still notice any problems, please contact us.
+
+As we found out earlier this month, there will be a power outage in Hearst Gym during the weekend of December 27-28 and January 3-4.
+
+Normally, this would result in all services being completely unavailable. However, we've put in a lot of effort to reduce the impact by transferring as much content to an off-site server as possible. Here's a summary of what to expect:
+
+ +
+The main OCF website will be available, but the wiki will not. Other services (like SSH, F/OSS mirrors, etc.) will be unavailable.
+
+
+
+We've spent a lot of time trying to minimize the impact of the power outage, but there are some things we can't do (we're extremely limited by the university's policies on off-site hosing, and our own lack of resources).
+
+
+
+If you have any questions, you can email us at help@ocf.berkeley.edu; we'll be able to view and respond to mail during the outage.
+ diff --git a/docs/news/posts/blogger-post-2015-01-14-1.md b/docs/news/posts/blogger-post-2015-01-14-1.md new file mode 100644 index 0000000..376d06f --- /dev/null +++ b/docs/news/posts/blogger-post-2015-01-14-1.md @@ -0,0 +1,8 @@ +--- +date: 2015-01-14 +--- + +# Downtime Monday for kernel updates +The login server (SSH) will be restarted Monday (Jan 19) night to apply security and performance updates. Total downtime should be less than 10 minutes.
+
+Edit: Originally, the downtime was only intended for the login (SSH) server. We're expanding it to include all servers to include recent security updates. Total downtime should be less than 20 minutes. diff --git a/docs/news/posts/blogger-post-2015-01-27-1.md b/docs/news/posts/blogger-post-2015-01-27-1.md new file mode 100644 index 0000000..b4417ae --- /dev/null +++ b/docs/news/posts/blogger-post-2015-01-27-1.md @@ -0,0 +1,6 @@ +--- +date: 2015-01-27 +--- + +# Downtime Tuesday for security updates +All servers will be restarted the night of Tuesday, Jan 27 to apply security updates. Sorry for the inconvenience. diff --git a/docs/news/posts/blogger-post-2015-01-29-1.md b/docs/news/posts/blogger-post-2015-01-29-1.md new file mode 100644 index 0000000..605d023 --- /dev/null +++ b/docs/news/posts/blogger-post-2015-01-29-1.md @@ -0,0 +1,6 @@ +--- +date: 2015-01-29 +--- + +# Firewall maintenance Feb. 17 +IST will be performing maintenance on OCF's firewall on Tuesday 2/17 from 5:30am to 7am. OCF services may be unavailable during this window. update: IST rescheduled the maintenance to 2/17 diff --git a/docs/news/posts/blogger-post-2015-02-12-1.md b/docs/news/posts/blogger-post-2015-02-12-1.md new file mode 100644 index 0000000..fa522b7 --- /dev/null +++ b/docs/news/posts/blogger-post-2015-02-12-1.md @@ -0,0 +1,6 @@ +--- +date: 2015-02-12 +--- + +# Downtime Sunday for maintenance +On Sunday night (February 15), we will be performing one-time maintenance on the OCF file server. Total downtime should be no more than two hours (and probably much less). diff --git a/docs/news/posts/blogger-post-2015-05-05-1.md b/docs/news/posts/blogger-post-2015-05-05-1.md new file mode 100644 index 0000000..d70c735 --- /dev/null +++ b/docs/news/posts/blogger-post-2015-05-05-1.md @@ -0,0 +1,12 @@ +--- +date: 2015-05-05 +--- + +# Network downtime (resolved) +Around 1:15am Tuesday morning, we starting experiencing high latency on our internal network. The high latency resulted in NFS reads/writes blocking for periods of several seconds, causing a backlog of processes on the web server and other servers. This resulted in timeouts when trying to access web pages, and eventually complete downtime when we took the servers offline.
+
+We had four different volunteer staff in the lab troubleshooting the issue around 1:30am. It was difficult to pin down because the actual cause was intermittent, so downtime was slightly more than 30 minutes. (We tried various steps such as searching for network loops, removing different servers from the network, disconnecting from campus, etc.)
+
+The ultimate cause was a broken backup script run by one of the student groups we host. From what we can tell, a daily backup script they had scheduled exceeded their disk quota, then continued thrashing the network trying to write blocks (which failed after exceeding the disk quota).
+
+We're monitoring the network now to ensure everything continues to operate normally, and will work on methods for limiting individual accounts' ability to cripple the network. We'll also improve our ability to monitor the network (our existing tools weren't granular enough for us to see the problem without directly witnessing it in iotop). diff --git a/docs/news/posts/blogger-post-2015-05-09-1.md b/docs/news/posts/blogger-post-2015-05-09-1.md new file mode 100644 index 0000000..ce3a366 --- /dev/null +++ b/docs/news/posts/blogger-post-2015-05-09-1.md @@ -0,0 +1,16 @@ +--- +date: 2015-05-09 +--- + +# Another WordPress XSS vulnerability; please update! +
+Another vulnerability was recently discovered in WordPress which affects a large number of OCF web hosting users. The vulnerability can potentially allow a malicious person to hijack your session and compromise your website.

All users should update immediately to the latest version of WordPress. Version 4.2.2 (i.e. the latest version) is the only version we consider safe.

Updating WordPress is extremely easy; it's just a single click after logging in to the admin panel.

Recent versions of WordPress come with automatic updates enabled for minor releases, which can help to protect you from future vulnerabilities. We strongly recommend not disabling this feature!

If we've contacted you and you need help updating your site, please don't hesitate to get in touch so that we can help!
+
+
+
+We will be emailing affected users in the near future and offering to upgrade WordPress on their behalf. If you'd like us to not do this, please confirm that either (a) you have updated it yourself, (b) you've removed WordPress entirely, or (c) you'd like to close your OCF account.
+
+
+
+Thanks for your help!
+ diff --git a/docs/news/posts/blogger-post-2015-05-20-1.md b/docs/news/posts/blogger-post-2015-05-20-1.md new file mode 100644 index 0000000..c8ba06e --- /dev/null +++ b/docs/news/posts/blogger-post-2015-05-20-1.md @@ -0,0 +1,6 @@ +--- +date: 2015-05-20 +--- + +# Directory listings disabled by default May 31th +On May 31th, we will disable Apache directory listings by default for both virtual hosts and userdir web hosting. You can re-enable these by creating a file named .htaccess in your web root with the line "Options +Indexes". diff --git a/docs/news/posts/blogger-post-2015-05-20-2.md b/docs/news/posts/blogger-post-2015-05-20-2.md new file mode 100644 index 0000000..7126bef --- /dev/null +++ b/docs/news/posts/blogger-post-2015-05-20-2.md @@ -0,0 +1,6 @@ +--- +date: 2015-05-20 +--- + +# Server maintenance 5/24 +We will be updating our physical servers on Sunday, May 24th around 9pm PDT. All OCF services will be affected, though we expect downtime to be less than 15 minutes. diff --git a/docs/news/posts/blogger-post-2015-05-20-3.md b/docs/news/posts/blogger-post-2015-05-20-3.md new file mode 100644 index 0000000..70e9f64 --- /dev/null +++ b/docs/news/posts/blogger-post-2015-05-20-3.md @@ -0,0 +1,6 @@ +--- +date: 2015-05-20 +--- + +# jaws maintenance 5/20 +We will be performing updates on jaws during Thursday, May 20th. jaws is a testing machine which hosts no public services, though staff VMs (and any services they provide) will be unavailable. diff --git a/docs/news/posts/blogger-post-2015-05-21-1.md b/docs/news/posts/blogger-post-2015-05-21-1.md new file mode 100644 index 0000000..e83f70a --- /dev/null +++ b/docs/news/posts/blogger-post-2015-05-21-1.md @@ -0,0 +1,6 @@ +--- +date: 2015-05-21 +--- + +# Online account tools maintenance tonight +The online account tools (used for requesting accounts, changing passwords, etc.) will be offline for about an hour tonight as we rebuild the server. No other services will be affected; you can still change your password via SSH if desired. diff --git a/docs/news/posts/blogger-post-2015-05-21-2.md b/docs/news/posts/blogger-post-2015-05-21-2.md new file mode 100644 index 0000000..596093c --- /dev/null +++ b/docs/news/posts/blogger-post-2015-05-21-2.md @@ -0,0 +1,14 @@ +--- +date: 2015-05-21 +--- + +# Mail server maintenance 5/22 +Our internal mail server (anthrax) will be unavailable for about an hour tomorrow. This only affects individual accounts with email forwarding, and mail sent from the OCF. Virtual hosts with email forwarding will still work.
+
+
+No mail will be lost; instead, it will be queued until the server is available again. At the worst, your mail may be delayed by an hour or two.
+
+
+
+Staff alumni with Google Apps accounts won't be affected at all.
+ diff --git a/docs/news/posts/blogger-post-2015-06-15-1.md b/docs/news/posts/blogger-post-2015-06-15-1.md new file mode 100644 index 0000000..4f77454 --- /dev/null +++ b/docs/news/posts/blogger-post-2015-06-15-1.md @@ -0,0 +1,8 @@ +--- +date: 2015-06-15 +--- + +# Server maintenance 6/19 +
+We will be updating our physical servers on Friday, June 19th around 9pm PDT. All OCF services will be affected, though we expect downtime to be less than 15 minutes.
+ diff --git a/docs/news/posts/blogger-post-2015-06-15-2.md b/docs/news/posts/blogger-post-2015-06-15-2.md new file mode 100644 index 0000000..6f8819c --- /dev/null +++ b/docs/news/posts/blogger-post-2015-06-15-2.md @@ -0,0 +1,6 @@ +--- +date: 2015-06-15 +--- + +# Directory listings re-enabled by default June 19th +On June 19th, we will re-enable Apache directory listings by default for both virtual hosts and userdir web hosting. You can disable these by creating a file named .htaccess in your web root with the line "Options -Indexes". diff --git a/docs/news/posts/blogger-post-2015-06-20-1.md b/docs/news/posts/blogger-post-2015-06-20-1.md new file mode 100644 index 0000000..420eb48 --- /dev/null +++ b/docs/news/posts/blogger-post-2015-06-20-1.md @@ -0,0 +1,34 @@ +--- +date: 2015-06-20 +--- + +# Ongoing downtime due to server crash +At 1:07pm today, hal, our primary production server, froze. We are on-site and working to restore it. We are moving the important servers to another machine while we investigate, as hal continues to experience issues.
+
+Update 2:07pm: Service has been restored, but we are continuing to move servers to a different machine. There will be some downtime as we continue the migration, but it will affect single services only.
+
+Remaining to migrate: (updated 5:04pm)
+
+ +
+
 +Update 5:20pm: All VMs are migrated to jaws, and all services should be restored. We'll be debugging and rebuilding hal in the near future, and will be scheduling downtime some time in the next few weeks to move VMs back. We'll post a followup here when we have a date in mind.
+ diff --git a/docs/news/posts/blogger-post-2015-08-09-1.md b/docs/news/posts/blogger-post-2015-08-09-1.md new file mode 100644 index 0000000..facc98a --- /dev/null +++ b/docs/news/posts/blogger-post-2015-08-09-1.md @@ -0,0 +1,10 @@ +--- +date: 2015-08-09 +--- + +# Downtime Wednesday for security updates +
+All servers will be restarted the night of Wednesday, August 12th to apply security updates. Sorry for the inconvenience.
+
 +Update: The downtime took longer than expected when our file server failed to come back up within the ~5 minutes we expect for a restart. It took an additional ~30 minutes to run a fsck. We'll be more careful to watch for and schedule these at more convenient times in the future.
+ diff --git a/docs/news/posts/blogger-post-2015-08-30-1.md b/docs/news/posts/blogger-post-2015-08-30-1.md new file mode 100644 index 0000000..92123d6 --- /dev/null +++ b/docs/news/posts/blogger-post-2015-08-30-1.md @@ -0,0 +1,14 @@ +--- +date: 2015-08-30 +--- + +# Removing automatic redirects from www.ocf.b.e/user -> www.ocf.b.e/~user +We are removing the automatic redirects from www.ocf.b.e/user -> www.ocf.b.e/~user.
+
+
+There appears to be virtually no traffic to these, so you should not notice any changes. ocf.io/user will continue to work.
+
+
+
+This will take effect in one week (Sept. 6).
+ diff --git a/docs/news/posts/blogger-post-2015-09-19-1.md b/docs/news/posts/blogger-post-2015-09-19-1.md new file mode 100644 index 0000000..f520720 --- /dev/null +++ b/docs/news/posts/blogger-post-2015-09-19-1.md @@ -0,0 +1,17 @@ +--- +date: 2015-09-19 +--- + +# Lab closed, some services affected by campus-wide network downtime +There was a fire in the campus datacenter a few hours ago which took down CalNet and related campus websites. See systemstatus.berkeley.edu for more details.
+
 +The following were affected by the CalNet outage, but are no longer affected
+
+ +
+WiFi access in the lab is still down. All OCF services themselves are fine.
+
 +The Publications Center made the decision to close the lab due to the WiFi outage. It will reopen once WiFi has been restored. diff --git a/docs/news/posts/blogger-post-2015-09-20-1.md b/docs/news/posts/blogger-post-2015-09-20-1.md new file mode 100644 index 0000000..9ae26bb --- /dev/null +++ b/docs/news/posts/blogger-post-2015-09-20-1.md @@ -0,0 +1,12 @@ +--- +date: 2015-09-20 +--- + +# Service outage due to hanging server reboot (resolved) +Public-facing services are down. Web hosting and SSH access have been down since approximately 2am PDT. The login server (tsunami) was restarted at 10:21pm.
+
+Staff are investigating the outage.
+
+Update 3:13am: Services are now recovering and should be fully restored within a few minutes.
+
+Update 3:18am: All services should now be restored. Please contact help@ocf.berkeley.edu if you are still encountering trouble. diff --git a/docs/news/posts/blogger-post-2015-10-09-1.md b/docs/news/posts/blogger-post-2015-10-09-1.md new file mode 100644 index 0000000..c9eb962 --- /dev/null +++ b/docs/news/posts/blogger-post-2015-10-09-1.md @@ -0,0 +1,6 @@ +--- +date: 2015-10-09 +--- + +# Scheduled outage Tuesday 10/13 from 7am to 7:30am +There will be about 30 minutes of network downtime on Tuesday, October 13th from 7am to 7:30am while IST performs firewall maintenance. All OCF services will be unavailable during this time. diff --git a/docs/news/posts/blogger-post-2015-10-12-1.md b/docs/news/posts/blogger-post-2015-10-12-1.md new file mode 100644 index 0000000..57c9a25 --- /dev/null +++ b/docs/news/posts/blogger-post-2015-10-12-1.md @@ -0,0 +1,20 @@ +--- +date: 2015-10-12 +--- + +# Lab hours have been extended +Starting Monday, October 12th, the Open Computing Facility lab's hours have been extended.
+
+They are now:
+ +
+See the lab page for a full list of hours and holidays.
+
+
+
+Many thanks to the Publications Center staff for extending the lab's hours!
+ diff --git a/docs/news/posts/blogger-post-2015-10-18-1.md b/docs/news/posts/blogger-post-2015-10-18-1.md new file mode 100644 index 0000000..68be7c6 --- /dev/null +++ b/docs/news/posts/blogger-post-2015-10-18-1.md @@ -0,0 +1,6 @@ +--- +date: 2015-10-18 +--- + +# Migrating some service-providing hosts +Over the next couple hours we'll be migrating some hosts to a different physical server. Individual services will have downtime of no more than a few minutes. diff --git a/docs/news/posts/blogger-post-2015-11-06-1.md b/docs/news/posts/blogger-post-2015-11-06-1.md new file mode 100644 index 0000000..634a261 --- /dev/null +++ b/docs/news/posts/blogger-post-2015-11-06-1.md @@ -0,0 +1,6 @@ +--- +date: 2015-11-06 +--- + +# Downtime tonight in preparation for lab move +There will be a few minutes of network downtime tonight after 9pm. This is in preparation for our move to MLK next week. diff --git a/docs/news/posts/blogger-post-2015-11-09-1.md b/docs/news/posts/blogger-post-2015-11-09-1.md new file mode 100644 index 0000000..c2faa20 --- /dev/null +++ b/docs/news/posts/blogger-post-2015-11-09-1.md @@ -0,0 +1,10 @@ +--- +date: 2015-11-09 +--- + +# We're moving to MLK Student Union! +
+OCF is moving! We're closing early this Thursday (at 6pm), and reopening next Monday, November 16th, in 171 MLK Student Union!
+
+Check out https://ocf.io/mlk for more information.
+ diff --git a/docs/news/posts/blogger-post-2015-11-09-2.md b/docs/news/posts/blogger-post-2015-11-09-2.md new file mode 100644 index 0000000..d41ab0e --- /dev/null +++ b/docs/news/posts/blogger-post-2015-11-09-2.md @@ -0,0 +1,8 @@ +--- +date: 2015-11-09 +--- + +# Downtime Monday night, Wednesday during day +There will be some downtime Monday night (after 9pm) possible into Tuesday morning for network maintenance in preparation for our move.
+
+There will be additional downtime Wednesday during the day (no more than an hour) as we move servers from our current location (Hearst Gym) to our new location (MLK Student Union). diff --git a/docs/news/posts/blogger-post-2015-11-14-1.md b/docs/news/posts/blogger-post-2015-11-14-1.md new file mode 100644 index 0000000..8150804 --- /dev/null +++ b/docs/news/posts/blogger-post-2015-11-14-1.md @@ -0,0 +1,6 @@ +--- +date: 2015-11-14 +--- + +# Unscheduled network downtime +As of 12:05am on 2015-11-14, the network is down unexpectedly. We are investigating. diff --git a/docs/news/posts/blogger-post-2015-12-04-1.md b/docs/news/posts/blogger-post-2015-12-04-1.md new file mode 100644 index 0000000..32ab38b --- /dev/null +++ b/docs/news/posts/blogger-post-2015-12-04-1.md @@ -0,0 +1,6 @@ +--- +date: 2015-12-04 +--- + +# Scheduled downtime Saturday night +We plan to perform maintenance on Saturday, December 5th around 10pm. All services will be unavailable during this time. Total downtime should be less than 20 minutes. diff --git a/docs/news/posts/blogger-post-2015-12-09-1.md b/docs/news/posts/blogger-post-2015-12-09-1.md new file mode 100644 index 0000000..61d170f --- /dev/null +++ b/docs/news/posts/blogger-post-2015-12-09-1.md @@ -0,0 +1,6 @@ +--- +date: 2015-12-09 +--- + +# Disk quota increased to 5 GB +Our Board of Directors approved an increase in disk quota from 2 GB to 5 GB, and it's now been implemented. Happy file storing! diff --git a/docs/news/posts/blogger-post-2015-12-17-1.md b/docs/news/posts/blogger-post-2015-12-17-1.md new file mode 100644 index 0000000..67d55d9 --- /dev/null +++ b/docs/news/posts/blogger-post-2015-12-17-1.md @@ -0,0 +1,7 @@ +--- +date: 2015-12-17 +--- + +# Scheduled downtime Thursday night to fsck disks +Due to a flurry of I/O errors on one of our physical servers during a routine update tonight, we're scheduling downtime tomorrow (Thursday) night after 9pm to fsck all virtual servers.
+
The total downtime should be no more than an hour and will have rolling outages. Key services like web hosting will only be unavailable for maybe 15 minutes. diff --git a/docs/news/posts/blogger-post-2016-01-19-1.md b/docs/news/posts/blogger-post-2016-01-19-1.md new file mode 100644 index 0000000..6130205 --- /dev/null +++ b/docs/news/posts/blogger-post-2016-01-19-1.md @@ -0,0 +1,8 @@ +--- +date: 2016-01-19 +--- + +# Username length limit raised to 16 characters +The maximum username length has been raised from 8 to 16 characters. Enjoy your longer usernames!
+
+Note that some commands, such as ps and w, may truncate usernames longer than 8 characters or do other strange behaviors. If you have any scripts which use such commands or otherwise assume usernames are 8 characters or fewer, beware that they may have to be revised. diff --git a/docs/news/posts/blogger-post-2016-01-21-1.md b/docs/news/posts/blogger-post-2016-01-21-1.md new file mode 100644 index 0000000..1fa6303 --- /dev/null +++ b/docs/news/posts/blogger-post-2016-01-21-1.md @@ -0,0 +1,8 @@ +--- +date: 2016-01-21 +--- + +# Scheduled downtime Thursday night +
+We plan to perform maintenance on Thursday, January 21st around 9pm. All services will be unavailable during this time. Total downtime should be less than 20 minutes
+ diff --git a/docs/news/posts/blogger-post-2016-01-22-1.md b/docs/news/posts/blogger-post-2016-01-22-1.md new file mode 100644 index 0000000..63cb736 --- /dev/null +++ b/docs/news/posts/blogger-post-2016-01-22-1.md @@ -0,0 +1,8 @@ +--- +date: 2016-01-22 +--- + +# Degraded internet access (resolved) +The OCF is currently experiencing degraded internet. We are investigating.
+
+Update: This has been resolved since 4:34pm. diff --git a/docs/news/posts/blogger-post-2016-02-01-1.md b/docs/news/posts/blogger-post-2016-02-01-1.md new file mode 100644 index 0000000..2ba1679 --- /dev/null +++ b/docs/news/posts/blogger-post-2016-02-01-1.md @@ -0,0 +1,13 @@ +--- +date: 2016-02-01 +--- + +# Campus internet access degraded (resolved) +The UC Berkeley campus is currently experiencing highly degraded internet (50+ ms latency and 30+% packet loss).
+
+
+
+This is affecting access to all OCF servers from outside of campus. It is also affecting AirBears, ResComp, EECS machines, and other campus resources.
+
+Update 4:12pm This appears to have resolved itself.
+ diff --git a/docs/news/posts/blogger-post-2016-02-05-1.md b/docs/news/posts/blogger-post-2016-02-05-1.md new file mode 100644 index 0000000..0ce73a8 --- /dev/null +++ b/docs/news/posts/blogger-post-2016-02-05-1.md @@ -0,0 +1,9 @@ +--- +date: 2016-02-05 +--- + +# Unexpected downtime Feb. 5 +There was around half an hour of unexpected downtime for all OCF web services from 12:15-12:45am on February 5 as a server had to be recovered from an accidental configuration. If you recognize any issues in the next few days, please report them to help@ocf.berkeley.edu.
+
+Thanks,
+OCF Staff diff --git a/docs/news/posts/blogger-post-2016-02-06-1.md b/docs/news/posts/blogger-post-2016-02-06-1.md new file mode 100644 index 0000000..55d51f7 --- /dev/null +++ b/docs/news/posts/blogger-post-2016-02-06-1.md @@ -0,0 +1,33 @@ +--- +date: 2016-02-06 +--- + +# Upgrading user-facing servers to jessie +In the past year we've upgraded our entire infrastructure to Debian jessie, with the exception of user-facing machines.
+
+The time to upgrade them is now. We've prepared upgraded versions of each of these servers and will swap them out early morning on Wednesday, Feb. 10th.
+
+The servers that will be upgraded are:
+
+ +
+Most users won't notice the update, except that most software will be a newer versions. The one exception is users who have dynamically-linked binaries somewhere in their home directories.
+
+Because many libraries will be upgrading, most of these programs will fail to run after the upgrade. The best solution is to recompile the binaries (or find newer, pre-compiled versions).
+
+One specific case is with environment managers like Python's virtualenv, Ruby's rbenv or rvm, and Node's nodeenv or nvm. These often put fully-compiled versions of the interpreter in your home directory, and in most cases, this will fail to work. After the upgrade, you'll need to rebuild these.
+
+For application hosting, you can find instructions on our website:
+https://www.ocf.berkeley.edu/docs/services/webapps/
+
+During the server swap, you should expect a small amount of downtime (about 5 minutes).
+
+If you have any questions or need assistance feel free to reach out to help@ocf.berkeley.edu.
+
+Update Feb 07: We're going to push this back until early morning Wednesday (originally it was Monday) to give us a little more time to ensure a smooth upgrade.
+
+Update Feb 09: For biohazard (app hosting), we'll be reaching out to individual groups using the server to coordinate a smooth upgrade. biohazard will continue to be available (and unupgraded); we'll be moving groups one-by-one to the new server (named werewolves). diff --git a/docs/news/posts/blogger-post-2016-02-12-1.md b/docs/news/posts/blogger-post-2016-02-12-1.md new file mode 100644 index 0000000..fa2ab82 --- /dev/null +++ b/docs/news/posts/blogger-post-2016-02-12-1.md @@ -0,0 +1,8 @@ +--- +date: 2016-02-12 +--- + +# MySQL downtime Feb 11 to fsck disks +MySQL will be offline for about 10 minutes tonight for emergency maintenance.
+
+This is in response to unscheduled downtime about an hour ago due to a kernel deadlock which took down all MySQL services. We don't really think it could be caused by filesystem corruption, but because of recent corruption at the OCF which affected nearly all servers (caused by Debian #788062) we think it's worth checking. diff --git a/docs/news/posts/blogger-post-2016-02-13-1.md b/docs/news/posts/blogger-post-2016-02-13-1.md new file mode 100644 index 0000000..89642c7 --- /dev/null +++ b/docs/news/posts/blogger-post-2016-02-13-1.md @@ -0,0 +1,14 @@ +--- +date: 2016-02-13 +--- + +# Downtime Saturday 2/20 for NFS migration +On Friday, Feb 19 we'll be migrating our NFS and production servers onto newer hardware backed by SSDs and more RAM.
+
+This will involve a couple hours of downtime (but hopefully not too many). We'll try to keep it quick.
+
+It will take place in the evening (10pm or later).
+
+Update 2/19: We're pushing this back until Saturday, Feb 20 evening.
+
+Update 2/20 11:50pm: This is done after about 30 minutes in read-only mode and about two minutes minutes of actual downtime. Some VMs will be moving over to jaws now to increase their performance, but the bulk of the work is done and you shouldn't notice any more issues. diff --git a/docs/news/posts/blogger-post-2016-02-16-1.md b/docs/news/posts/blogger-post-2016-02-16-1.md new file mode 100644 index 0000000..81b4850 --- /dev/null +++ b/docs/news/posts/blogger-post-2016-02-16-1.md @@ -0,0 +1,6 @@ +--- +date: 2016-02-16 +--- + +# Downtime tonight to apply security updates +There will be about 20 minutes of downtime tonight to apply security updates. diff --git a/docs/news/posts/blogger-post-2016-02-29-1.md b/docs/news/posts/blogger-post-2016-02-29-1.md new file mode 100644 index 0000000..50bcbf8 --- /dev/null +++ b/docs/news/posts/blogger-post-2016-02-29-1.md @@ -0,0 +1,12 @@ +--- +date: 2016-02-29 +--- + +# mirrors.ocf.berkeley.edu read-only for about 20 hours +We're moving mirrors.ocf.berkeley.edu, our free-and-open-source software mirror, from its current hardware (a recycled desktop with some extra hard drives) onto a new server (with server-grade harddrives, RAID, etc.).
+
+To do this with the minimum amount of downtime, we're going to be copying the disk from our current mirror to the new server. To ensure consistency, we need to first make it read-only. We expect the copy to take about 8 hours, after which point we'll make the replacement server the main mirror. At this point, mirrors will be about 8 hours old, but will quickly catch back up when the cronjobs start running.
+
+Update 8:30pm 2/28: This is starting now.
+
+Update 5:33pm 2/29: Maintenance is complete. diff --git a/docs/news/posts/blogger-post-2016-03-11-1.md b/docs/news/posts/blogger-post-2016-03-11-1.md new file mode 100644 index 0000000..d3969ef --- /dev/null +++ b/docs/news/posts/blogger-post-2016-03-11-1.md @@ -0,0 +1,6 @@ +--- +date: 2016-03-11 +--- + +# Downtime Friday 3/11 for security updates +There will be about 20 minutes of downtime Friday night to apply security updates. diff --git a/docs/news/posts/blogger-post-2016-03-11-2.md b/docs/news/posts/blogger-post-2016-03-11-2.md new file mode 100644 index 0000000..5e3b6e6 --- /dev/null +++ b/docs/news/posts/blogger-post-2016-03-11-2.md @@ -0,0 +1,29 @@ +--- +date: 2016-03-11 +--- + +# MySQL and printing unavailable for 34 minutes (resolved) +MySQL and printing were unavailable today for about 34 minutes due to an unscheduled outage.
+
+Why it was down
+For background, all of the OCF's production infrastructure is supposed to live on two physical servers: jaws and pandemic. There's a third legacy physical server named hal which hosts some testing machines and our backups.
+
+Due to a problem removing a backup logical volume which led to a deadlock and many processes in uninterruptable sleep after Tuesday (believed to be a kernel bug), a staff member gave 15 minutes warning before restarting hal today (Thursday) to try to fix the issue. Since hal isn't supposed to hold important services, this should be totally safe and is considered an acceptable warning period, since normally the only people who will even notice are other staff.
+
+Unfortunately, two servers, pollution (the print server) and maelstrom (the MySQL server) were on hal due to some temporary migrations. They should have been moved back to jaws about a week ago, but weren't.
+
+When hal went down, it took down these production services, killing MySQL (which also took down many websites, the OCF's website, Request Tracker, ...) and printing in the lab. This was realized as soon as monitoring triggered, and the staff member phoned another staffer currently in the lab after hal wasn't coming back up.
+
+Due to a misconfiguration, hal entered maintenance mode, and the other staffer had to enter the root password and fix the filesystem configuration before hal would boot. As soon as hal booted, MySQL and printing were started and service was restored.
+
+Timeline
+
+ + diff --git a/docs/news/posts/blogger-post-2016-03-19-1.md b/docs/news/posts/blogger-post-2016-03-19-1.md new file mode 100644 index 0000000..f1eeb6d --- /dev/null +++ b/docs/news/posts/blogger-post-2016-03-19-1.md @@ -0,0 +1,37 @@ +--- +date: 2016-03-19 +--- + +# MySQL read-only Saturday 3/19 +As part of our work to transition from Percona to MariaDB for our MySQL server, we'll be migrating user data tonight around 9pm.
+
+
+
+To do this, we'll put the existing Percona server into read-only mode, then make a final import to the new MariaDB host. We believe this will take about an hour and don't anticipate any issues (we've already tested imports from our regular backups without problems).
+
+
+
+Read-only mode is necessary during the import to ensure we get a consistent backup, and so that writes made during the transition are not lost.
+
+
+
+Some sites may experience downtime while the server is in read-only mode (if they require writing to the database to show pages). Most sites will experience some level of degradation (e.g. can't log in to admin or edit posts).
+
+Update 10:01pm: We have entered read-only mode.
+
+Update 10:10pm: The backup is complete and is being imported into MariaDB now.
+
+Update 10:14pm: ETA 35 minutes.
+
+Update 10:35pm: The import was interrupted when the new server ran out of memory. We're increasing memory / reducing memory use by mysqld and starting the import again. Still in read-only mode.
+
+Update 10:45pm: ETA 38 minutes.
+
+Update 11:11pm: Import has finished, we're now swapping out MariaDB for Percona (which will involve about 2 minutes of downtime).
+
+Update 11:16pm: We've noticed some issues with the import (views were not correctly copied) so we'll need to re-do the import. Still in read-only mode, expect another hour or two in this state. Sorry for the trouble!
+
+Update 11:44pm: The view problem is fixed, so we're proceeding to move MariaDB into production. Expect about 2-3 minutes of downtime now.
+
+Update 11:55pm: All work is completed and we are now on MariaDB. Total downtime was about 3 minutes, with read-only mode lasting about two hours.
+ diff --git a/docs/news/posts/blogger-post-2016-03-29-1.md b/docs/news/posts/blogger-post-2016-03-29-1.md new file mode 100644 index 0000000..da8f3ec --- /dev/null +++ b/docs/news/posts/blogger-post-2016-03-29-1.md @@ -0,0 +1,6 @@ +--- +date: 2016-03-29 +--- + +# Short webserver downtime 3/29 for maintenance +We'll be performing some short maintenance on our webserver tonight (3/29) to install instrumentation that would let us diagnose the kernel panics we've been experiencing recently on it. diff --git a/docs/news/posts/blogger-post-2016-03-30-1.md b/docs/news/posts/blogger-post-2016-03-30-1.md new file mode 100644 index 0000000..ceb36fe --- /dev/null +++ b/docs/news/posts/blogger-post-2016-03-30-1.md @@ -0,0 +1,36 @@ +--- +date: 2016-03-30 +--- + +# Misleading Daily Cal article incorrectly suggests OCF connection to printer abuse +We were disappointed to find that the Daily Cal has published a misleading article about recent printer "hacking" at UC Berkeley.
+
+The online edition of the article contained (and still does) a photo of the OCF lab taken yesterday. The print version is even more misleading:
+
+
+
+
+The OCF was not involved in this attack in any way. Our printers are not exposed directly to the internet, and our volunteer staff take security very seriously. We were quite shocked to see ourselves on the Daily Cal article.
+
+It's really unfortunate that a lot of people will see this image and think that the OCF was compromised. Our volunteer staff have put a lot of work into the OCF, and we don't like to see it tarnished this way.
+
+We appreciate that the article itself did not reference the OCF. And we'd greatly appreciate help from the Daily Cal in replacing the misleading photo and issuing a clarification.
+
+Update March 30: The Daily Cal has updated the caption of the photo, but refuses to correct their mistake by replacing the photo, citing "a pretty strict policy on retraction".
+
+While we're glad they've recognized the mistake and added the clarifying caption, we still believe they should act with integrity to correct this mistake. It is not responsible journalism to place the photo of a widely recognized but completely unrelated organization above an article about poor security practices.
+
+Many people don't read Daily Cal articles in full, but instead see snippets posted on Facebook or elsewhere. In these snippets, a picture of the OCF is still prominently featured, and the work of our volunteer staff is still being devalued.
+
+
+
+
+
+
+
+This could have been prevented if the staff of the Daily Cal had taken a few minutes to shoot us a quick email prior to publishing that article. A lot of damage has already been done. The Daily Cal should take responsibility for their error and correct it.
+
+
+
+Update April 1: The Daily Cal has now replaced the photo with one of a printer which was actually affected by the attack. We thank them for correcting the mistake.
+ diff --git a/docs/news/posts/blogger-post-2016-04-15-1.md b/docs/news/posts/blogger-post-2016-04-15-1.md new file mode 100644 index 0000000..390d862 --- /dev/null +++ b/docs/news/posts/blogger-post-2016-04-15-1.md @@ -0,0 +1,8 @@ +--- +date: 2016-04-15 +--- + +# Downtime tonight for hardware upgrades +There will be up to 45 minutes of downtime tonight for hardware upgrades. We don't expect to use the full window, but are reserving it in case of problems. It will begin some time around 10pm.
+
+Sorry for the inconvenience -- this should make future operations much more stable. diff --git a/docs/news/posts/blogger-post-2016-05-15-1.md b/docs/news/posts/blogger-post-2016-05-15-1.md new file mode 100644 index 0000000..eec526f --- /dev/null +++ b/docs/news/posts/blogger-post-2016-05-15-1.md @@ -0,0 +1,10 @@ +--- +date: 2016-05-15 +--- + +# Home directories read-only Sunday for a few minutes +We are in the process of upgrading our NFS storage. To do this, we'll need to make home directories read-only for a few minutes on Sunday afternoon, then once again late Sunday evening.
+
+We anticipate the read-only mode to last only about 15 minutes, but there may be an additional 5-10 minutes of full downtime while we transition into it.
+
+Most websites should continue to work during read-only mode. SSH will continue to work, but with limited capabilities. diff --git a/docs/news/posts/blogger-post-2016-05-26-1.md b/docs/news/posts/blogger-post-2016-05-26-1.md new file mode 100644 index 0000000..fba7090 --- /dev/null +++ b/docs/news/posts/blogger-post-2016-05-26-1.md @@ -0,0 +1,8 @@ +--- +date: 2016-05-26 +--- + +# Power outage, services down +At 9:35am, a power outage in MLK caused our servers to go down. We are working to bring most services back up by using power from a different circuit. We are in contact with facilities to restore power.
+
+Update 1:03pm: All services should be restored. diff --git a/docs/news/posts/blogger-post-2016-07-18-1.md b/docs/news/posts/blogger-post-2016-07-18-1.md new file mode 100644 index 0000000..05770ca --- /dev/null +++ b/docs/news/posts/blogger-post-2016-07-18-1.md @@ -0,0 +1,28 @@ +--- +date: 2016-07-18 +--- + +# Introducing HTTPS for virtual hosts +The staff of the OCF strongly believe in the need for encryption of everyday communication. This case has been made not just by invasive government spying, but also by wireless carriers (like Verizon) maliciously modifying requests to increase ad revenue.We think that the internet needs to transition to encrypting all communications whenever possible.
+
+Two years ago, we transitioned all of www.ocf.berkeley.edu (all regular user websites) to HTTPS-only. At the time, it was impractical to also transition virtual hosts, because there was no way for us to acquire the necessary SSL certificates without a lot of manual effort.
+
+With the creation of Let's Encrypt, a free provider of SSL certificates that prioritizes automation, we're finally able to start offering HTTPS for virtually-host websites as well. Starting today, we'll be slowly rolling out HTTPS for our virtual hosts. We expect the full roll-out to complete within about 3 months.
+
+What do I need to do?
+Most likely, nothing. We'll automatically switch your site over to HTTPS when we acquire a certificate for it. (We're unable to immediately acquire certificates for all of our ~500 virtual hosts due to Let's Encrypt's rate limits.)
+
+Will this break existing links to my website? Do I need to update posters with the new link?
+No. We will issue 301 redirects to the updated URL.
+
+Are you sending the HSTS header?
+Not yet, but we'd like to in the future. We want to make sure we've tested the current setup for some time first, as once we start sending the header, we can never go back to plain HTTP.
+
+We do send the HSTS header for www.ocf.berkeley.edu (user websites).
+
+My site broke after the switch to HTTPS, what do I do?
+It's unlikely, but possible, that some sites will break because of this. Typically, this is because of the wrong site URL set within a CMS like WordPress. You should be able to fix this by logging in to the admin panel and making sure your site URL has https:// at the front.
+
+If there's anything we can do to help, don't hesitate to send us an email.
+
+Update 2016-10-09: The roll-out is complete. diff --git a/docs/news/posts/blogger-post-2016-08-24-1.md b/docs/news/posts/blogger-post-2016-08-24-1.md new file mode 100644 index 0000000..5c6bad4 --- /dev/null +++ b/docs/news/posts/blogger-post-2016-08-24-1.md @@ -0,0 +1,22 @@ +--- +date: 2016-08-24 +--- + +# Changes to our lab hours +Our hours of operation this semester have undergone some changes from last year's. Without any ado, our tentative weekly hours for Fall 2016 are
+
+Monday: 9am-6pm
+Tuesday: 9am-8pm
+Wednesday: 9am-8pm
+Thursday: 9am-8pm
+Friday: 9am-5pm
+Saturday: 12pm-5pm
+Sunday: 12pm-5pm
+
+These hours are still subject to change. Specifically, once our Board of Directors meeting time is finalized, Monday's hours may be extended to 8pm and we will then close at 6pm on a different weekday. Remember, you can always see an up-to-date listing of our regular hours and our hours for the next seven days at https://ocf.io/lab.
+
+The most noticeable difference between last year's hours and this semester's is that we have had to scale back our weekend hours, Friday through Sunday. The OCF does not take lightly that this change is a setback for the accessibility of our lab to our members, but this is the most agreeable outcome we can offer. When scaling back our hours, we were very careful to select the times which would ultimately affect the least number of users of the lab.
+
+Nonetheless, we are proud to be reopening at 9am Wednesday morning for another collaborative and productive fall semester. If you have any questions, comments, or concerns about our new hours, please drop by one of our upcoming BoD meetings and let your voice be known.
+
+Thanks for flying OCF! diff --git a/docs/news/posts/blogger-post-2016-08-26-1.md b/docs/news/posts/blogger-post-2016-08-26-1.md new file mode 100644 index 0000000..15ca1af --- /dev/null +++ b/docs/news/posts/blogger-post-2016-08-26-1.md @@ -0,0 +1,8 @@ +--- +date: 2016-08-26 +--- + +# Downtime tonight (8/26) for kernel updates +There will be a brief period of downtime tonight (8/26), around 9 pm, likely lasting about 20-45 minutes, as we apply kernel updates to our servers and run some hardware checks. Our software mirrors may be down for more time, as that server has a potential hardware issue and could require longer to check.
+
+Sorry for the inconvenience and contact us if you have any questions about our services or the upcoming downtime! diff --git a/docs/news/posts/blogger-post-2016-09-16-1.md b/docs/news/posts/blogger-post-2016-09-16-1.md new file mode 100644 index 0000000..ba046a5 --- /dev/null +++ b/docs/news/posts/blogger-post-2016-09-16-1.md @@ -0,0 +1,14 @@ +--- +date: 2016-09-16 +--- + +# Weak user passwords have been reset recently +We're always looking for ways to make sure our users' accounts remain secure. It's important to have strong passwords because OCF accounts aren't just used for lab access or printing, but also for web hosting and many other services. These services are accessible via the internet, which means anybody (including those with no relation to UC Berkeley) could try to guess the passwords.
+
+Recently, we attempted to guess the passwords of our users in an attempt to find users with exceptionally weak passwords and have them upgrade to a stronger password. To do this, we used both lists of passwords from online password dumps (including passwords used by real people on other services), lists of the most common passwords, and other methods such as trying different variations of each character of a password.
+
+This is the same thing an attacker would do to try to break into a user's account. It's important to keep in mind that we cannot read users' passwords. We store them in a non-reversible manner as password hashes. Passwords are hashed using a one-way function that can be computed at login and compared to the stored hash. At no time are OCF staff able to find out the passwords of our users, except in the way above (by trying hundreds of thousands of passwords until one matches).
+
+The guessing method only works on the weakest of passwords; it also explains why the best way to make a password strong is to make it long, rather than trying to make it more complicated by adding numbers or special characters.
+
+We have removed the passwords from the accounts we were able to guess passwords for. These users should either reset their password online (if they have a CalNet account linked to their OCF account, as most current and recent students do), or use the manual verification process otherwise. diff --git a/docs/news/posts/blogger-post-2016-09-16-2.md b/docs/news/posts/blogger-post-2016-09-16-2.md new file mode 100644 index 0000000..eb22cba --- /dev/null +++ b/docs/news/posts/blogger-post-2016-09-16-2.md @@ -0,0 +1,10 @@ +--- +date: 2016-09-16 +--- + +# Ongoing lab closures due to understaffing +The OCF has already had to announce one reduction in our hours of operation this semester, and we're sorry to say that further closures on top of that will be the norm for the next several weeks. Anyone who has visited the lab or our website in the last two weeks has noticed that we are missing a couple hours on Thursday and Friday from what was previously announced, and that, for the first time in recent memory, we are closed all day every Saturday.
+
+While we would like to say there was something we can do about this, it is unfortunately out of our control: the Publications and Media Center is responsible for providing the paid employees who man the front desk during all hours the lab is open. Unfortunately, the PMC has been operating with a reduced staff this semester and is still seeking to hire replacements. We must conclude that these additional closures will continue for a few more weeks at least.
+
+The OCF is not happy about the state of things, but the hiring process takes time, and there is not any way for us to speed things up. In the meantime, please keep an eye on our hours page; as much as possible, if we can predict when we will be closed, we will update the hours on our website. Please also check our homepage, as we will make an announcement there as soon as possible in the event of an unexpected closure during the day. diff --git a/docs/news/posts/blogger-post-2016-09-20-1.md b/docs/news/posts/blogger-post-2016-09-20-1.md new file mode 100644 index 0000000..f840bd3 --- /dev/null +++ b/docs/news/posts/blogger-post-2016-09-20-1.md @@ -0,0 +1,16 @@ +--- +date: 2016-09-20 +--- + +# Taking applications for new work-study position +UPDATE: The application is closed! Thanks to all who applied!
+
+The OCF is looking for someone to fulfill the role of OCF Operations Strategist, a new, hired position developed by OCF staff and the first known paid position in OCF history. The Operations Strategist will work closely with the OCF throughout the Fall 2016/Spring 2017 academic year to help guide the OCF as it undergoes the major transition of becoming a fully self-sustained organization starting next year.
+
+For the last several years, the OCF has relied on the support of paid Publications and Media Center staff to open and close on time, provide security, and carry out day-to-day operational tasks when volunteer staff cannot attend the lab. This arrangement is coming to a close next year, however, and thus the OCF is currently fully committed to acquire funding and develop the strategies we need for the front desk to become 100% staffed by OCF hires when the time comes. This new position provided by the PMC will ensure a smooth transition.
+
+The Operations Strategist will play a crucial part during the entire process, and so will need to be committed to the position for the entire academic year. Moreover, the Operations Strategist is not just a front desk job. While performing desk tasks will be a minor part of the Operations Strategist's role, the great majority of the 12-15 hour work week will be dedicated to projects vital to the OCF's transition. We are seeking someone with creative problem-solving skills who can work autonomously on solutions to problems we will face in hiring and managing our own team to staff the lab.
+
+The link to the application is here: http://bit.ly/2deW0ry. Unfortunately, we are only considering applicants who have work-study funds to last the entire academic year.
+
+Interviews are next week. diff --git a/docs/news/posts/blogger-post-2016-09-22-1.md b/docs/news/posts/blogger-post-2016-09-22-1.md new file mode 100644 index 0000000..4372571 --- /dev/null +++ b/docs/news/posts/blogger-post-2016-09-22-1.md @@ -0,0 +1,10 @@ +--- +date: 2016-09-22 +--- + +# Power outage 7am-9am Sunday 9/25 +All OCF services will be unavailable for a few hours this Sunday morning due to a scheduled power outage in the MLK Student Union. There will be no access to SSH and websites will be down for the duration.
+
+We will be shutting down servers around 6:30am and hope to be back online by 9:30am.
+
+UPDATE: Power was restored by 9:30am. All services should be back online as of 10:35am. diff --git a/docs/news/posts/blogger-post-2016-10-01-1.md b/docs/news/posts/blogger-post-2016-10-01-1.md new file mode 100644 index 0000000..28a97c2 --- /dev/null +++ b/docs/news/posts/blogger-post-2016-10-01-1.md @@ -0,0 +1,37 @@ +--- +date: 2016-10-01 +--- + +# Announcing revamped mail hosting for student groups +We're excited to announce the launch of our new mail service for student groups!
+
+The OCF has provided mail virtual hosting for many years, allowing student groups to create email addresses such as "{name}@mygroup.berkeley.edu" which forward to their primary email address.
+
+With the new system, we're introducing a few new features:
+
+ +
+You can find out more about using the new system on our website.
+
+What if my student group already had mail virtual hosting?
+
+We've automatically migrated all of your existing forwarding addresses to the new system; there's nothing you need to do. In the future, you'll use the new web-based interface for modifying addresses instead of the .forward files.
+
+Can I forward to multiple addresses?
+
+Yup, each address can forward to as many other addresses as you'd like!
+
+Can I have a catch-all address at my domain?
+
+Yes, and the web interface will help you set this up.
+
+How do I get mail virtual hosting for my group?
+
+If you already have virtual hosting, then we can easily add mail service. If not, we can help you set up both a website and mail hosting.
+
+For details on our web virtual hosting service, see here.
+
+For details on adding mail virtual hosting, see here. diff --git a/docs/news/posts/blogger-post-2016-10-20-1.md b/docs/news/posts/blogger-post-2016-10-20-1.md new file mode 100644 index 0000000..9c46f4a --- /dev/null +++ b/docs/news/posts/blogger-post-2016-10-20-1.md @@ -0,0 +1,6 @@ +--- +date: 2016-10-20 +--- + +# Downtime tonight (10/20) for security updates +There will be a brief period of downtime tonight (10/20), around 10 pm, likely lasting about 20-45 minutes, as we apply kernel updates to our servers. diff --git a/docs/news/posts/blogger-post-2016-11-07-1.md b/docs/news/posts/blogger-post-2016-11-07-1.md new file mode 100644 index 0000000..fa6bee8 --- /dev/null +++ b/docs/news/posts/blogger-post-2016-11-07-1.md @@ -0,0 +1,8 @@ +--- +date: 2016-11-07 +--- + +# Changes to account policies +We've revamped our account policies, principally to be more specific on the reasons accounts can be disabled. We've also established a formal procedure for revoking the membership privileges of disruptive OCF members.
+
+Our revised account policies are accessible at https://www.ocf.berkeley.edu/docs/services/account/account-policies/, and the banning policy is accessible at https://www.ocf.berkeley.edu/docs/membership/banning/. diff --git a/docs/news/posts/blogger-post-2016-11-08-1.md b/docs/news/posts/blogger-post-2016-11-08-1.md new file mode 100644 index 0000000..5b1153f --- /dev/null +++ b/docs/news/posts/blogger-post-2016-11-08-1.md @@ -0,0 +1,7 @@ +--- +date: 2016-11-08 +--- + +# Proposed Constitutional Changes +There will be a general meeting on Monday, November 14 at 7:10 PM in the OCF lab to discuss a proposed new OCF Constitution. The text of the proposal is at https://www.ocf.berkeley.edu/~kpengboy/ConstitutionBill2016.html.
+All are welcome to come and contribute their perspectives. diff --git a/docs/news/posts/blogger-post-2016-11-21-1.md b/docs/news/posts/blogger-post-2016-11-21-1.md new file mode 100644 index 0000000..a11ffe5 --- /dev/null +++ b/docs/news/posts/blogger-post-2016-11-21-1.md @@ -0,0 +1,12 @@ +--- +date: 2016-11-21 +--- + +# Scheduled power outage Tuesday 11/22 4am-8am +All OCF services will be unavailable for a few hours on Tuesday morning due to a scheduled power outage in the MLK Student Union. There will be no access to SSH and websites will be down for the duration.
+
+We will be shutting down servers around 3:30am and hope to be back online by 8:30am.
+
 +There is also a chance of additional power outages until 3:00pm.
+
+EDIT: As of 9:19am, user-facing services have been restored. diff --git a/docs/news/posts/blogger-post-2016-12-03-1.md b/docs/news/posts/blogger-post-2016-12-03-1.md new file mode 100644 index 0000000..2d1bed4 --- /dev/null +++ b/docs/news/posts/blogger-post-2016-12-03-1.md @@ -0,0 +1,8 @@ +--- +date: 2016-12-03 +--- + +# Daily print quota change 12/5-12/16 +For all you who need to print out final papers, the OCF daily printing quota will be increased to 20 pages/day for RRR week and finals week this semester.
+
+This is not a permanent change, nor have we decided yet whether to repeat this bump in future semesters. The semesterly quota is still 100 pages. diff --git a/docs/news/posts/blogger-post-2017-02-18-1.md b/docs/news/posts/blogger-post-2017-02-18-1.md new file mode 100644 index 0000000..2b0bc67 --- /dev/null +++ b/docs/news/posts/blogger-post-2017-02-18-1.md @@ -0,0 +1,23 @@ +--- +date: 2017-02-18 +--- + +# MySQL read-only Saturday 2/18 +As part of our work to transition from jessie to stretch for our MySQL server, we'll be migrating user data today around 5pm.
+
+To do this, we'll put the existing jessie server into read-only mode, then make a final import to the new stretch host. We believe this will take about an hour and don't anticipate any issues (we've already tested imports from our regular backups without problems).
+
+Read-only mode is necessary during the import to ensure we get a consistent backup, and so that writes made during the transition are not lost.
+
+Some sites may experience downtime while the server is in read-only mode (if they require writing to the database to show pages). Most sites will experience some level of degradation (e.g. can't log in to admin or edit posts).
+
+Update 5:01pm: We have entered read-only mode.
+Update 5:58pm: The backup has completed, we are now importing it into the new database server. Probably another 45 minutes or so.
+Update 6:07pm: ETA 25 minutes.
+Update 6:15pm: ETA 16 minutes
+Update 6:25pm: Unfortunately we hit a snag during the import and have to start it over.
+Update 6:29pm: ETA 37 minutes.
+Update 6:40pm: ETA 21 minutes.
+Update 6:46pm: ETA 10 minutes.
+Update 6:56pm: Import has finished, we're now switching back into rw mode. This will involve a few seconds of downtime.
+Update 6:59pm: MySQL is now available as usual. Total time in read-only mode was about two hours. Total downtime was less than a minute. diff --git a/docs/news/posts/blogger-post-2017-02-19-1.md b/docs/news/posts/blogger-post-2017-02-19-1.md new file mode 100644 index 0000000..ccfd8b6 --- /dev/null +++ b/docs/news/posts/blogger-post-2017-02-19-1.md @@ -0,0 +1,14 @@ +--- +date: 2017-02-19 +--- + +# LDAP/Kerberos upgrade Sunday 2/19 +As part of our ongoing work to transition from jessie to stretch at the OCF, we will be migrating our LDAP/Kerberos server today starting around 7:20 PM.
+
+To do this, we'll import LDAP and Kerberos data from our on-site backups to test that they still work. Although they have been tested before, it's always nice to make sure that backups actually work and include the right data. After importing these backups, a final dump and sync will be made in case any data has changed in the small amount of time that it takes to import the backup data.
+
 +All users will be unable to log in while the migration is taking place, but this should only be a very short time, as the majority of the migration will be done before the final sync is done.
+
 +Contact us if you have any questions or concerns about the upgrade!
+
 +Update 7:40: All users should be able to log in again, please let us know if you have any issues with accessing your account! diff --git a/docs/news/posts/blogger-post-2017-03-17-1.md b/docs/news/posts/blogger-post-2017-03-17-1.md new file mode 100644 index 0000000..0927a4a --- /dev/null +++ b/docs/news/posts/blogger-post-2017-03-17-1.md @@ -0,0 +1,10 @@ +--- +date: 2017-03-17 +--- + +# Mirrors disk migration 3/17 +There will be a short period of downtime between 10 AM and 12 PM (noon) today (3/17) to migrate the disks containing our software mirrors from one server to another.
+
+Update 10:18: Our mirroring server is down, and the disks are being migrated.
+
+Update 10:34: Our mirroring server is back up with its disks full migrated! Let us know if you encounter any problems. diff --git a/docs/news/posts/blogger-post-2017-04-20-1.md b/docs/news/posts/blogger-post-2017-04-20-1.md new file mode 100644 index 0000000..1ace41e --- /dev/null +++ b/docs/news/posts/blogger-post-2017-04-20-1.md @@ -0,0 +1,8 @@ +--- +date: 2017-04-20 +--- + +# Downtime this morning - Thursday 4/20 6:53 to 11:15 AM +From around 6:53 to 11:15 this morning a majority of the OCF's services were down, including websites, email forwarding, software mirrors, etc. We take downtime issues like this very seriously, so we are still in the process of doing a full postmortem on exactly what happened and the timeline it followed. Expect that to be posted soon as we finalize the details of what happened, but everything should now be back up and working again. If you notice any problems, please contact us and we'd be happy to help.
+
+Thanks as always for flying OCF! diff --git a/docs/news/posts/blogger-post-2017-05-31-1.md b/docs/news/posts/blogger-post-2017-05-31-1.md new file mode 100644 index 0000000..c58ba93 --- /dev/null +++ b/docs/news/posts/blogger-post-2017-05-31-1.md @@ -0,0 +1,28 @@ +--- +date: 2017-05-31 +--- + +# Announcing HTTPS for web apps +Last summer, we introduced automatic HTTPS for all OCF-hosted websites with virtual hosting, helping keep private data sent to these websites safe from government spies and would-be password thieves. Today, we are announcing the same feature for our web application hosting service.
+
+This enhancement has already been rolled out for existing web apps, and it will come out-of-the-box for new ones. For more info, check out the original announcement as well as the updated FAQ below.
+
+

+What do I need to do?

+Most likely, nothing. Thanks to greatly increased Let's Encrypt request rate limits, we can download certificates and configure your website to use SSL almost immediately.
+
+

+Will this break existing links to my website? Do I need to update posters with the new link?

+No. We will issue 301 redirects to the updated URL.
+
+

+Are you sending the HSTS header?

+Still not yet, but it's certainly on the agenda now that we've been using HTTPS on regular virtual hosts for almost a year. We already send the HSTS header for www.ocf.berkeley.edu, including all websites for individual accounts.
+
+

+What even is app hosting? How do I get it?

+Web application hosting is a relatively new service we offer which allows groups to develop sophisticated websites using modern web technologies not available with regular virtual hosting. The eligibility requirements are the same as for regular virtual hosting, and you can apply be emailing hostmaster@ocf.berkeley.edu. Please read the help page for more information.
+
+If you have any more questions, please email us.
+
+Finally, you can log into your website at ease! diff --git a/docs/news/posts/blogger-post-2017-07-08-1.md b/docs/news/posts/blogger-post-2017-07-08-1.md new file mode 100644 index 0000000..d1e49dc --- /dev/null +++ b/docs/news/posts/blogger-post-2017-07-08-1.md @@ -0,0 +1,6 @@ +--- +date: 2017-07-08 +--- + +# Downtime for security updates evening of 7/8 +There will be a brief period of downtime tonight (July 8th, 2017), around 9 PM, likely lasting about 15-30 minutes. We plan to apply kernel updates to our servers for recent security vulnerabilities and upgrade one of our servers to the most recent version of Debian stable (stretch). Let us know if you have any questions or concerns and thanks for flying OCF! diff --git a/docs/news/posts/blogger-post-2017-08-02-1.md b/docs/news/posts/blogger-post-2017-08-02-1.md new file mode 100644 index 0000000..762766f --- /dev/null +++ b/docs/news/posts/blogger-post-2017-08-02-1.md @@ -0,0 +1,11 @@ +--- +date: 2017-08-02 +--- + +# Expected power outage 8/2 +Due to a wildfire, the UC Berkeley campus is expecting a power outage around 2:40 PM today. All OCF services have been powered down in preparation.
+
 + +We apologize for the inconvenience.
+
 +Update (9:02 PM): services have been restored. diff --git a/docs/news/posts/blogger-post-2017-10-15-1.md b/docs/news/posts/blogger-post-2017-10-15-1.md new file mode 100644 index 0000000..b7ab57d --- /dev/null +++ b/docs/news/posts/blogger-post-2017-10-15-1.md @@ -0,0 +1,6 @@ +--- +date: 2017-10-15 +--- + +# New: bulk-add vhost mail addresses +Starting 10/15, you can bulk-add forwarding addresses for mail virtual hosting by uploading a CSV file. diff --git a/docs/news/posts/blogger-post-2017-12-24-1.md b/docs/news/posts/blogger-post-2017-12-24-1.md new file mode 100644 index 0000000..c31a237 --- /dev/null +++ b/docs/news/posts/blogger-post-2017-12-24-1.md @@ -0,0 +1,6 @@ +--- +date: 2017-12-24 +--- + +# Downtime scheduled for afternoon of 12/24 +Tomorrow afternoon between 3-5pm the OCF expects to undergo a short period of downtime to reboot our physical servers in order to apply security updates. We do not expect our services to be unavailable for longer than 30 minutes. Let us know if you have any questions or concerns. Thank you for flying OCF and have a happy holiday season! diff --git a/docs/news/posts/blogger-post-2018-01-17-1.md b/docs/news/posts/blogger-post-2018-01-17-1.md new file mode 100644 index 0000000..bdefb7b --- /dev/null +++ b/docs/news/posts/blogger-post-2018-01-17-1.md @@ -0,0 +1,20 @@ +--- +date: 2018-01-17 +--- + +# Downtime on 1/17/18 +We're currently recovering from a multi-drive RAID failure on one of our servers. Currently logins, MySQL, printing, account creation, and our internal DNS will all not work. We'll post status updates as we bring things back up.
+
+Downtime was on 1/17 from 13:27 to around 18:00 in total.
+
+Note: MySQL and accounts created are being rolled back to a backup as of 2 AM this morning (1/17), so any accounts created since then or database changes will be rolled back.
+
+Update 1/17 15:25: MySQL, logins, and DNS are all in the process of being restored. Database restoration will likely take around another hour, perhaps more, but logins and DNS are both close to being fixed.
+
+Update 1/17 16:22: Logins (LDAP/Kerberos) and DNS have been restored, MySQL is getting closer to being restored, probably will be about half an hour on restoring that. Printing is still a work in progress, but getting closer. Account creation is still down.
+
+Update 1/17 18:09: Everything except account creation should be back up again. We are working on restoring all accounts created today and then we will re-enable account creation. MySQL databases have been restored from our most recent backup (2 AM this morning). We have noticed some lingering issues with DNS that we will keep checking and update here if it stays a problem. Let us know if you encounter any other problems with our services!
+
+Update 1/17 19:58: Account creation has been re-enabled, although any new accounts made today still need to be re-created, but new accounts made from this point on should work fine. We also fixed a DNSSEC issue that was preventing valid DNS responses in some cases. Contact us if you notice any other problems!
+
+Update 1/18 03:21: All the missing accounts have been re-created again, but they will need to reset their passwords, since we purposefully do not log those for security and they were lost in our Kerberos rollback. These accounts can reset their passwords through our website. diff --git a/docs/news/posts/blogger-post-2018-02-10-1.md b/docs/news/posts/blogger-post-2018-02-10-1.md new file mode 100644 index 0000000..50110aa --- /dev/null +++ b/docs/news/posts/blogger-post-2018-02-10-1.md @@ -0,0 +1,8 @@ +--- +date: 2018-02-10 +--- + +# NFS downtime on 2018-02-09 +NFS (containing home directories, cron jobs, and public_html web directories) was down for a short period of time on 2018-02-09 from 4:41 PM to around 5:06 PM. In this time, files stored in user home directories and all hosted websites (including https://www.ocf.berkeley.edu) were unavailable. Logins were also affected, both in the OCF lab and from outside the lab to our SSH server. Thank you for your patience while we brought everything back online.
+
+Please contact us if you have any questions or concerns! diff --git a/docs/news/posts/blogger-post-2018-03-20-1.md b/docs/news/posts/blogger-post-2018-03-20-1.md new file mode 100644 index 0000000..c8e4a88 --- /dev/null +++ b/docs/news/posts/blogger-post-2018-03-20-1.md @@ -0,0 +1,11 @@ +--- +date: 2018-03-20 +--- + +# Short downtime tonight (3/19) for security updates +There will be a brief period of downtime tonight (3/19/2018), around 8:15 PM. It will likely last about 30-45 minutes as we restart our servers to apply kernel updates to them.
+
 +Edit (8:20 PM): We have started. See you on the flip side!
+Edit (8:40 PM): We have rebooted a majority of our public services, they should be coming up soon. Public mirrors are still to come.
+Edit (8:50 PM): Our MySQL database did not start correctly, so we have started that now. We are also in the process of rebooting our public software mirrors host.
+Edit (9:00 PM): Restarting our servers is fully complete, thanks for your patience! diff --git a/docs/news/posts/blogger-post-2018-06-02-1.md b/docs/news/posts/blogger-post-2018-06-02-1.md new file mode 100644 index 0000000..3e60284 --- /dev/null +++ b/docs/news/posts/blogger-post-2018-06-02-1.md @@ -0,0 +1,15 @@ +--- +date: 2018-06-02 +--- + +# Small amount of downtime today (6/1) +We had a short amount of downtime today with a few of our services between 10:10 PM and 11:19 PM today (6/1/2018) due to some networking issues on one of our hypervisors:
+
+ +
+We have fully recovered all affected services and put a couple protections in place to avoid this kind of issue again.
+ diff --git a/docs/news/posts/blogger-post-2018-10-04-1.md b/docs/news/posts/blogger-post-2018-10-04-1.md new file mode 100644 index 0000000..7d0d4d6 --- /dev/null +++ b/docs/news/posts/blogger-post-2018-10-04-1.md @@ -0,0 +1,15 @@ +--- +date: 2018-10-04 +--- + +# Downtime on October 6 +The OCF will be experiencing downtime, due to scheduled maintenance, on October 6th, from 9PM-12AM. Hosted websites will experience downtime as we briefly reboot the servers to apply critical security updates.  Once our servers are rebooted, users accessing our public login server and our apphosting server will not be able to write files as we are moving NFS to a new host. This read-only period will last no longer than 15 minutes and all operations should behave as normal by midnight at the latest.
+
+Thanks for flying with the OCF!
+
+8:53 Update: We've powered off the servers to do networking hardware and kernel updates.
+11:03 Update: We've decided not to do the NFS migration tonight, but networking updates have been performed and services should be back up in the next hour.
+11:42 Update: Most of our public servers should be back now, apart from our public mirrors and our own website (vhosts should be fine).
+12:17 Update: Our website is back, but our software mirrors are not back yet.
+12:51 Update: Everything should be back and working now except our HPC control server, which we are still debugging.
+1:35 Update: This is the all clear, everything should be working now! Feel free to let us know by emailing help@ocf.berkeley.edu if you notice anything wrong. diff --git a/docs/news/posts/blogger-post-2018-10-08-1.md b/docs/news/posts/blogger-post-2018-10-08-1.md new file mode 100644 index 0000000..ef03db1 --- /dev/null +++ b/docs/news/posts/blogger-post-2018-10-08-1.md @@ -0,0 +1,16 @@ +--- +date: 2018-10-08 +--- + +# IPv6 Connectivity Issues on October 8 +Starting last night, the OCF has been experiencing some connectivity issues to our public SSH server over IPv6. If you are having trouble logging in to ssh.ocf.berkeley.edu, please try using IPv4 to connect. To do this, you can add -4 to your SSH connection command, like so
+
+ssh -4 @ssh.ocf.berkeley.edu
+
+If your SSH client does not support the -4 flag, you can also connect directly to our server's IPv4 address. To do this, just connect to `169.229.226.25` instead of `ssh.ocf.berkeley.edu`.
+
+Some other services such as MySQL may also experience issues over IPv6. If neither IPv6 nor IPv4 is working for you, please let us know.
+
+Thank you for being patient while we restore full connectivity.
+
+UPDATE 2018-10-09 1:45AM: IPv6 connectivity should be restored to all our user-facing services. diff --git a/docs/news/posts/blogger-post-2018-10-12-1.md b/docs/news/posts/blogger-post-2018-10-12-1.md new file mode 100644 index 0000000..0833c1b --- /dev/null +++ b/docs/news/posts/blogger-post-2018-10-12-1.md @@ -0,0 +1,8 @@ +--- +date: 2018-10-12 +--- + +# Scheduled maintenance on night of 2018-10-12 +The OCF is anticipating a short period of intermittent service unavailability in order to perform some additional maintenance on our hypervisors as a followup to last week's maintenance event. Specifically, we intend to migrate NFS to our new fileserver, reinstall our hypervisors onto new disks, and possibly migrate our mirrors server to new hardware. We are scheduling this event for for low-utilization hours at night to minimize any disruption to our users.
+
+Thanks for flying OCF and send us an email if you have any questions! diff --git a/docs/news/posts/blogger-post-2018-10-12-2.md b/docs/news/posts/blogger-post-2018-10-12-2.md new file mode 100644 index 0000000..1c25b50 --- /dev/null +++ b/docs/news/posts/blogger-post-2018-10-12-2.md @@ -0,0 +1,6 @@ +--- +date: 2018-10-12 +--- + +# Account Creation and Password Resets Temporarily Down, October 12 +Due to ongoing maintenance, account creation and password resets are down today.

At roughly 6:15PM, we there will be brief NFS downtime as we attempt to fix the issue.

Thanks for flying OCF!
diff --git a/docs/news/posts/blogger-post-2018-11-07-1.md b/docs/news/posts/blogger-post-2018-11-07-1.md new file mode 100644 index 0000000..a8dc15a --- /dev/null +++ b/docs/news/posts/blogger-post-2018-11-07-1.md @@ -0,0 +1,37 @@ +--- +date: 2018-11-07 +--- + +# Upgrading user-facing servers to Debian stretch +In the past year and a half we've upgraded our entire infrastructure to Debian stretch, with the notable exception of any user-facing machines.
+
+Edit (2018-11-25): We've now upgraded all of the servers listed below! Please contact help@ocf.berkeley.edu if you have any questions about the upgrade or notice anything broken.
+
+The time to upgrade them is now! We've prepared upgraded versions of each of these servers and will swap them out on Sunday, November 18th, 2018 the evening of Sunday, November 25th, 2018. This has been postponed from the 18th since some apphosting groups did not have their applications ready on the new server to make the migration possible.
+
+The servers that will be upgraded are:
+ +Most users won't notice the update, except that most software will have newer versions. The one main exception is users who have dynamically-linked binaries somewhere in their home directories.
+
+Because many libraries will be upgrading, most of these kinds of programs will fail to run after the upgrade. The best solution is to recompile the binaries (or find newer, pre-compiled versions).
+
+One specific case is with environment managers like Python's virtualenv, Ruby's rbenv/rvm, and Node's nodeenv/nvm. These often put fully-compiled versions of the interpreter in your home directory, and in most cases, these will fail to work once the server is upgraded. After the upgrade, you'll need to rebuild these to get them to work again. Here are some major versions of programs that will be upgrading:
+
+- Ruby 2.1.5 -> Ruby 2.3.3
+- Python 3.4.2 -> Python 3.5.3
+- Python 2.7.9 -> Python 2.7.13
+- NodeJS 0.10.29 -> NodeJS 4.8.2
+- PHP 5.6.36 -> PHP 7.0.30
+- Perl 5.20.2 -> Perl 5.24.1
+
+For upgrading any apps using our application hosting, you can find more detailed instructions on our website: https://www.ocf.berkeley.edu/docs/services/webapps/
+
+During the server swap, you should expect a small amount of downtime (about 5-10 minutes) as the new servers are swapped into place of the old servers.
+
+If you have any questions or need assistance feel free to reach out to help@ocf.berkeley.edu.
+
+Thanks for flying OCF! diff --git a/docs/news/posts/blogger-post-2018-12-21-1.md b/docs/news/posts/blogger-post-2018-12-21-1.md new file mode 100644 index 0000000..4691653 --- /dev/null +++ b/docs/news/posts/blogger-post-2018-12-21-1.md @@ -0,0 +1,8 @@ +--- +date: 2018-12-21 +--- + +# Downtime on 2018-12-20 +The OCF's web, application, and public login servers were unavailable from 10:43 PM PST to 11:21 PM PST tonight (2018-12-20) due to an NFS outage. The issue has since been resolved and we expect no further issues.
+
+Thanks for flying OCF and contact us if you have any questions or concerns! diff --git a/docs/news/posts/blogger-post-2019-01-04-1.md b/docs/news/posts/blogger-post-2019-01-04-1.md new file mode 100644 index 0000000..7113792 --- /dev/null +++ b/docs/news/posts/blogger-post-2019-01-04-1.md @@ -0,0 +1,17 @@ +--- +date: 2019-01-04 +--- + +# Introducing OCF PGP Keyserver Service +The OCF is happy to announce the public release of our PGP keyserver service, available at pgp.ocf.berkeley.edu. We are proud to be a part of the community supporting reliable public key infrastructure!
+
+At present, we are providing HKP and sks-recon on IPv4 and IPv6, as well as HTTP and HTTPS access. We are building out support for HKPS and keyserver pool membership. The following domains are supported, at the accompanying ports.
+ +We will soon be providing service at pgp.berkeley.edu and keyserver.berkeley.edu as well.
+
+If you would like to peer your SKS server instance with ours, please feel free to submit a pull request against our membership file! diff --git a/docs/news/posts/blogger-post-2019-05-03-1.md b/docs/news/posts/blogger-post-2019-05-03-1.md new file mode 100644 index 0000000..6334645 --- /dev/null +++ b/docs/news/posts/blogger-post-2019-05-03-1.md @@ -0,0 +1,16 @@ +--- +date: 2019-05-03 +--- + +# Introducing the OCF Mastodon Service! +
+
+We’re excited to announce that the OCF is now the proud host of its own Mastodon instance run by our volunteers! Many thanks to our hard-working volunteer staff for getting it set up and working on Kubernetes. Anyone with an OCF account can sign up for our instance with their OCF email (username@ocf.berkeley.edu). If you don’t have an OCF account and are eligible, make one for free!
+
 +What is Mastodon? 
+Mastodon is a free and open source federated social media platform. What does that mean? It’s kind of like email. Anyone can host their own Mastodon instance (like email servers) and talk to people on other instances (like a GMail user to a Yahoo! Mail user). This allows each instance to have its own moderation (in our case, it’s the OCF volunteer staff) while still being able to talk with the wider community. If you’re looking for an android app to browse Mastodon with, Tusky and Fedilab are both free and open source applications!
+
+
+
+
+ diff --git a/docs/news/posts/blogger-post-2019-05-23-1.md b/docs/news/posts/blogger-post-2019-05-23-1.md new file mode 100644 index 0000000..9b8aabe --- /dev/null +++ b/docs/news/posts/blogger-post-2019-05-23-1.md @@ -0,0 +1,14 @@ +--- +date: 2019-05-23 +--- + +# Downtime on May 21, 22:40 to 22:50 +Due to a misconfiguration of OCF's network switches related to changing settings for STP (Spanning Tree Protocol), the OCF experienced brief downtime from 22:40 to 22:50 PST on 2019-05-21.
+
+The issue has since been resolved and we expect no further issues.
+
 +
+We have completed an internal postmortem review of the incident to prevent and mitigate future issues.
+
+
+
Thanks for flying OCF and contact us if you have any questions or concerns! diff --git a/docs/news/posts/blogger-post-2019-06-14-1.md b/docs/news/posts/blogger-post-2019-06-14-1.md new file mode 100644 index 0000000..9848e28 --- /dev/null +++ b/docs/news/posts/blogger-post-2019-06-14-1.md @@ -0,0 +1,10 @@ +--- +date: 2019-06-14 +--- + +# Scheduled Maintenance on 2019-06-15 +On this Saturday, June 15 2019, between 7-10pm the OCF expects to undergo a short period of downtime to reboot our physical servers in order to apply security updates. We do not expect the maintenance to be longer than an hour. Let us know if you have any questions or concerns. Thanks for flying OCF!
+
+Update (10:30PM): although maintenance is progressing, it is taking slightly longer than expected. We hope to have primary services restored shortly.
+
+Update 2 (12:20AM): All primary services were restored around 11:30pm. diff --git a/docs/news/posts/blogger-post-2019-09-20-1.md b/docs/news/posts/blogger-post-2019-09-20-1.md new file mode 100644 index 0000000..a7977fc --- /dev/null +++ b/docs/news/posts/blogger-post-2019-09-20-1.md @@ -0,0 +1,9 @@ +--- +date: 2019-09-20 +--- + +# Scheduled Maintenance on 2019-09-21 +The OCF will have scheduled downtime from 9pm - 12am on Saturday 9/21. The hypervisors are scheduled to be restarted to apply security updates. We apologize for any inconvenience this may cause. If you have any issues resulting from the restart, please don’t hesitate to contact OCF staff at help@ocf.berkeley.edu.
+
+
+ diff --git a/docs/news/posts/blogger-post-2019-10-01-1.md b/docs/news/posts/blogger-post-2019-10-01-1.md new file mode 100644 index 0000000..d26e3a4 --- /dev/null +++ b/docs/news/posts/blogger-post-2019-10-01-1.md @@ -0,0 +1,8 @@ +--- +date: 2019-10-01 +--- + +# Increased Disk Quota and More Printing Pages +
+We have recently undergone an infrastructure upgrade and have doubled the amount of pages you can print per day, from 100 to 200 a semester and 10 to 20 per day!

We have also increased our Disk Quota, from 5 to 15 Gigabytes!
+ diff --git a/docs/news/posts/blogger-post-2019-10-09-1.md b/docs/news/posts/blogger-post-2019-10-09-1.md new file mode 100644 index 0000000..7087980 --- /dev/null +++ b/docs/news/posts/blogger-post-2019-10-09-1.md @@ -0,0 +1,17 @@ +--- +date: 2019-10-09 +--- + +# Downtime and Lab Closure due to PG&E Public Safety Power Shutdown +Campus has notified us that we are likely to lose power early Wednesday (10/9) as part of the PG&E Public Safety Power Shutdown (PSPS). The lab will be closed and all OCF services will be offline starting Tuesday night until power is restored. Campus has indicated that power may be off for 48 hours or more. We will be posting updates about lab closure and service status on this blog, so check back for updates.
+
+Update 10/8 10:50PM: We will be starting to shut down services at 1AM.
+
+Update 10/9 10:40PM: Since we did not lose power within the 8-10PM timeframe indicated tonight by PG&E, we are turning our most critical servers back on until we hear a definitive time. We're sorry that everything was down today without any actual power outage, but this was due to a conservative policy on our part and misleading and vague communication from PG&E. At this point, webhosting and email hosting should be working. If we lose power or it is clear that we will lose power soon, we will post another update here.
+
+Update 10/10 1:30AM: As campus power outage is supposedly imminent, we are shutting down servers again. We expect everything to be down until power is restored, excepting other surprises. The lab will be closed Thursday Oct. 10.
+
+Update 10/11 2:15PM: Power has been restored and we are beginning to power on the OCF servers. We hope to have all services functional as soon as possible. The lab will remain closed and will reopen tomorrow, Saturday Oct. 12 at the normal time, 9am.
+
+Update 10/11 3:00PM: All services should be available at this point. We are monitoring for any lingering problems. If you're having any trouble, send us an email at help@ocf.berkeley.edu.
+ diff --git a/docs/news/posts/blogger-post-2019-10-23-1.md b/docs/news/posts/blogger-post-2019-10-23-1.md new file mode 100644 index 0000000..2e260df --- /dev/null +++ b/docs/news/posts/blogger-post-2019-10-23-1.md @@ -0,0 +1,12 @@ +--- +date: 2019-10-23 +--- + +# Webhosting outage +Since about 6am this morning, webhosting has been unavailable. We are investigating the issue.
+
+Update 11:27am: University IST has blocked our server due to a user compromise. The issue has been resolved on our end and we are waiting for them to unblock us.
+
+Update 1:15pm: All services appear to be fully functional.
+
+Update 2:15pm: A lingering network misconfiguration resulted in downtime from aroximately 2:00pm - 2:10pm. The issue has been resolved. diff --git a/docs/news/posts/blogger-post-2019-10-28-1.md b/docs/news/posts/blogger-post-2019-10-28-1.md new file mode 100644 index 0000000..236090b --- /dev/null +++ b/docs/news/posts/blogger-post-2019-10-28-1.md @@ -0,0 +1,12 @@ +--- +date: 2019-10-28 +--- + +# PG&E PSPS not expected to impact service availability +Due to the Public Safety Power Shutoff, campus has moved power entirely to its local co-generation plant. Because of this, campus buildings, including MLK, are closed. The OCF Lab has been and will be closed throughout the duration of the outage. We do not know yet when we will reopen.
+
+Other OCF services, including webhosting and mail, are expected to be available throughout the PSPS as we hope to have co-generation power available. We have disabled our high performance computing cluster and turned off non-essential servers to reduce power consumption.
+
+We will post any further updates here.
+
+Update 10/28 11:30pm: The OCF Lab will reopen tomorrow Tuesday Oct 29 with our regular hours from 9am-8pm. diff --git a/docs/news/posts/blogger-post-2019-11-07-1.md b/docs/news/posts/blogger-post-2019-11-07-1.md new file mode 100644 index 0000000..b717038 --- /dev/null +++ b/docs/news/posts/blogger-post-2019-11-07-1.md @@ -0,0 +1,8 @@ +--- +date: 2019-11-07 +--- + +# Web Server Outage +Due to a compromise on our web server, our web server has been inaccessible since 8:00am 11/7/2019. We are working to resolve the issue.
+
+Update: The issue was resolved around 10:15am and everything appears to be working correctly. diff --git a/docs/news/posts/blogger-post-2019-11-12-1.md b/docs/news/posts/blogger-post-2019-11-12-1.md new file mode 100644 index 0000000..67540ee --- /dev/null +++ b/docs/news/posts/blogger-post-2019-11-12-1.md @@ -0,0 +1,6 @@ +--- +date: 2019-11-12 +--- + +# Student Group Account Management Outage +We are currently experiencing an outage in the API for CalLink, resulting in all users being unable to manage group accounts associated with student groups. More information will be provided later. diff --git a/docs/news/posts/blogger-post-2020-01-24-1.md b/docs/news/posts/blogger-post-2020-01-24-1.md new file mode 100644 index 0000000..1d2acb8 --- /dev/null +++ b/docs/news/posts/blogger-post-2020-01-24-1.md @@ -0,0 +1,13 @@ +--- +date: 2020-01-24 +--- + +# Major Service Outage +Due to a unexpected issue, certain OCF services are currently down, including the OCF website, account creation and Mastodon. Group web hosting is still functioning as expected.
+
+
+
+We will update this post as we have more information on the situation.
+
+Update (Jan 24, 12:46am): We have restored all service and will continue to monitor our services for issues.
+ diff --git a/docs/news/posts/blogger-post-2020-03-07-1.md b/docs/news/posts/blogger-post-2020-03-07-1.md new file mode 100644 index 0000000..1aabe26 --- /dev/null +++ b/docs/news/posts/blogger-post-2020-03-07-1.md @@ -0,0 +1,6 @@ +--- +date: 2020-03-07 +--- + +# Planned Service Downtime for 2020-03-013 +Around 10pm PST on March 13th (this Saturday), there will be scheduled downtime in order to migrate all of our servers over to an uninterruptible power supply. We expect this to take no longer than an hour, but will update this status blog if the situation changes. Apologies for the short notice. diff --git a/docs/news/posts/blogger-post-2020-03-10-1.md b/docs/news/posts/blogger-post-2020-03-10-1.md new file mode 100644 index 0000000..5fbe27a --- /dev/null +++ b/docs/news/posts/blogger-post-2020-03-10-1.md @@ -0,0 +1,13 @@ +--- +date: 2020-03-10 +--- + +# Extended Lab Closure +The OCF will be closed until further notice in the interest of public health.
+
Due to the large amount of traffic OCF gets during our opening hours, we have decided it would be best to close the lab to prevent possible spread of the coronavirus.
+
+We apologize for any inconvenience that this may cause, and if you are in urgent need of technology resources or printing, the following places provide computers and printers for use:

- Unit 1, 2, and 3 Academic Center
- Foothill Academic Center
- Clark Kerr Academic Center
- Moffitt Library
- Doe Library
- Jean Gray Hargrove Music Library
- Art History/Classics Library
- Environmental Design Library
- Physics Library
- and more
+
+OCF services besides the physical lab space, including webhosting and mail, are expected to remain available throughout the closure.
+
We will constantly evaluate the situation and reopen when it improves.

Thanks for Flying OCF!

- The OCF Staff
+
 diff --git a/docs/news/posts/blogger-post-2020-03-18-1.md b/docs/news/posts/blogger-post-2020-03-18-1.md new file mode 100644 index 0000000..17d97ee --- /dev/null +++ b/docs/news/posts/blogger-post-2020-03-18-1.md @@ -0,0 +1,13 @@ +--- +date: 2020-03-18 +--- + +# Major Service Outage +Many OCF services are currently unavailable, including the OCF website and personal/group web hosting services. +
+
+We are investigating the issue and will update this status post as we learn more.
+
+Update 10:10 AM: We have resolved the root problem and are gradually bringing services back up
+
+Update 10:48 AM: We believe all services are operational. We are continuing to monitor. If you're having issues, please send us an email at help@ocf.berkeley.edu. diff --git a/docs/news/posts/blogger-post-2020-04-03-1.md b/docs/news/posts/blogger-post-2020-04-03-1.md new file mode 100644 index 0000000..f58c39c --- /dev/null +++ b/docs/news/posts/blogger-post-2020-04-03-1.md @@ -0,0 +1,12 @@ +--- +date: 2020-04-03 +--- + +# Moderate Service Outage +The OCF is currently investigating a service outage that has disrupted various services such as our main website (ocfweb), ircbot, Keycloak, and more. All other services, such as web hosting and mirrors, should be functioning as normal.
+
+We will update this post once we have more information on the situation.
+
+Update 01:03 PM: Updated the scope of the outage to include ocfweb.
+
+Update 01:21 PM: The issue has been traced back to internal DNS issues and has now been resolved. We will continue to monitor our services for issues. diff --git a/docs/news/posts/blogger-post-2020-05-18-1.md b/docs/news/posts/blogger-post-2020-05-18-1.md new file mode 100644 index 0000000..66d487c --- /dev/null +++ b/docs/news/posts/blogger-post-2020-05-18-1.md @@ -0,0 +1,10 @@ +--- +date: 2020-05-18 +--- + +# Major service outage +The OCF is currently undergoing an unplanned reboot of our main hypervisor which may disrupt most if not all of our services. We expect to be back up shortly.
+
+Edit (19:08 PDT): Most services are back, but we're still working to stabilize all services and make sure services like HPC are fully working.
+
+Edit (19:22 PDT): All services should be back. Let us know at help@ocf.berkeley.edu if you notice that something is down. diff --git a/docs/news/posts/blogger-post-2020-07-04-1.md b/docs/news/posts/blogger-post-2020-07-04-1.md new file mode 100644 index 0000000..ebf081a --- /dev/null +++ b/docs/news/posts/blogger-post-2020-07-04-1.md @@ -0,0 +1,8 @@ +--- +date: 2020-07-04 +--- + +# RSO Group account creation/password reset broken due to expired signatory status +As of today, RSO signatory status for the previous school year has expired. This means if you were a signatory previously, you will no longer be able to create a new OCF group account for it or reset the password to an existing one using the web interface.
+
+We encourage you to re-register your RSO and add signatories as soon as possible, and are evaluating options depending on how long it takes for signatory status to be reinstated in CalLink especially with the turbulent current state of affairs. Most likely, we'll be forced to suspend account creation until we can verify your status as a signatory. If you need help resetting a password to an existing OCF group account contact help@ocf.berkeley.edu. diff --git a/docs/news/posts/blogger-post-2020-07-26-1.md b/docs/news/posts/blogger-post-2020-07-26-1.md new file mode 100644 index 0000000..8183e6b --- /dev/null +++ b/docs/news/posts/blogger-post-2020-07-26-1.md @@ -0,0 +1,10 @@ +--- +date: 2020-07-26 +--- + +# MySQL upgrade saturday 726 +As part of our work to transition from stretch to buster for our MySQL server, we'll be migrating user data today around 9:34pm.
+
+To do this, we'll be setting up replication on the primary onto a dev instance, which we will then switch the 'mysql' hostname over to. There will be a short cutover period once we switch over the host. This post will be updated if any issues arise.
+
+EDIT: The upgrade has been completed. We were under 10 minutes of read-only time and around 5 minutes of downtime. diff --git a/docs/news/posts/blogger-post-2020-09-17-1.md b/docs/news/posts/blogger-post-2020-09-17-1.md new file mode 100644 index 0000000..af1b198 --- /dev/null +++ b/docs/news/posts/blogger-post-2020-09-17-1.md @@ -0,0 +1,8 @@ +--- +date: 2020-09-17 +--- + +# Scheduled downtime for Sunday 9/20/20 from 12pm-5pm +

The current situation with COVID has caused us to fall behind on a number of software upgrades that are critical to keeping our systems running efficiently and securely, especially on our physical servers. Fortunately we've been able to secure a block of time on from the university in order to perform these upgrades. However, in order to upgrade our physical hypervisors it may be necessary to shut down some production virtual machines. We will try to limit downtime to as little as possible.

The following services (and possibly others) may be intermittently down during this time period (9/20/20 12-5pm):

The following services will be down for an extended period of time:

We suggest you do not schedule large jobs on HPC and have plans in place if +your site is expecting a large number of visitors during this time +period.

diff --git a/docs/news/posts/blogger-post-2020-09-28-1.md b/docs/news/posts/blogger-post-2020-09-28-1.md new file mode 100644 index 0000000..0415d2d --- /dev/null +++ b/docs/news/posts/blogger-post-2020-09-28-1.md @@ -0,0 +1,6 @@ +--- +date: 2020-09-28 +--- + +# Mirrors out of date due to server issues +

We're observing extended issues in syncing a number of our mirrors including Ubuntu and Kali, and so some of them may be very out of date.

We're currently working on resolving these issues and will update when we have more information.

Update: the issue has been resolved

diff --git a/docs/news/posts/blogger-post-2020-10-03-1.md b/docs/news/posts/blogger-post-2020-10-03-1.md new file mode 100644 index 0000000..ed5cf06 --- /dev/null +++ b/docs/news/posts/blogger-post-2020-10-03-1.md @@ -0,0 +1,6 @@ +--- +date: 2020-10-03 +--- + +# Mirrors down while we rebuild the RAID array +

 We are replacing a drive in our mirrors server so it is down. This is expected to be done by 2PM PST 3/10/20.

In addition, from 12:01AM on 05/10, we will temporarily shut down the mirrors service to allow the RAID array to rebuild properly. This may cause a prolonged outage.

EDIT: The rebuild is now complete. We're syncing the mirrors live so you might see some outdated packages.

diff --git a/docs/news/posts/blogger-post-2021-05-31-1.md b/docs/news/posts/blogger-post-2021-05-31-1.md new file mode 100644 index 0000000..f1b3516 --- /dev/null +++ b/docs/news/posts/blogger-post-2021-05-31-1.md @@ -0,0 +1,6 @@ +--- +date: 2021-05-31 +--- + +# Major web service outage +

The OCF is currently investigating an ongoing service outage that has disrupted web services including our main website (ocfweb) and user-hosted websites. All other services, such as our login servers (tsunami), HPC, and mirrors, should be functioning as normal.

Due to the timing of this outage, our response remains limited throughout the early night. We will update this post once we have more information on the situation.

diff --git a/docs/news/posts/blogger-post-2021-06-24-1.md b/docs/news/posts/blogger-post-2021-06-24-1.md new file mode 100644 index 0000000..7648515 --- /dev/null +++ b/docs/news/posts/blogger-post-2021-06-24-1.md @@ -0,0 +1,6 @@ +--- +date: 2021-06-24 +--- + +# Major Mirrors Outage +

 We are suffering from a significant RAID array failure of our mirrors server and thus will not be able to serve any of our mirrors traffic. We expect this down time to last for at least another week while we bring up a new server to serve mirrors.

 

We hope you can understand that this will take longer than we would hope due to limited site access and reduced staffing over the summer.

diff --git a/docs/news/posts/blogger-post-2021-07-05-1.md b/docs/news/posts/blogger-post-2021-07-05-1.md new file mode 100644 index 0000000..9f1b3ca --- /dev/null +++ b/docs/news/posts/blogger-post-2021-07-05-1.md @@ -0,0 +1,6 @@ +--- +date: 2021-07-05 +--- + +# Intermittent service outages during summer break +

The OCF is currently aware of intermittent service outages during the summer break to our OCF website, web hosting services, mailing services, mirrors, and internal infrastructure. Volunteer staffers have begun an investigation of the issue and are planning to resolve it in the coming weeks. Please expect delays in this process due to the limited number of staff available in the summer and other factors beyond our control. We will update this post once we have more information on the situation.

If you are a user or represent a group that has been adversely impacted by a recent OCF service outage, please do not hesitate to contact help@ocf.berkeley.edu so we can investigate further.

Update: This has been resolved.

diff --git a/docs/news/posts/blogger-post-2021-09-23-1.md b/docs/news/posts/blogger-post-2021-09-23-1.md new file mode 100644 index 0000000..6828aa4 --- /dev/null +++ b/docs/news/posts/blogger-post-2021-09-23-1.md @@ -0,0 +1,6 @@ +--- +date: 2021-09-23 +--- + +# Emergency Mirrors Maintenance +

2021-09-27 19:05 PST: mirrors are back up, this downtime should be resolved. If you notice any issues using our mirrors please reach out to us at help@ocf.berkeley.edu.


We are currently (9/22) running emergency maintenance on mirrors.ocf.berkeley.edu. Due to an earlier drive failure we've had to replace the drives. We expect the mirrors server to be down until Monday, but we will keep this blog post updated.

diff --git a/docs/news/posts/blogger-post-2021-10-08-1.md b/docs/news/posts/blogger-post-2021-10-08-1.md new file mode 100644 index 0000000..1f6b54c --- /dev/null +++ b/docs/news/posts/blogger-post-2021-10-08-1.md @@ -0,0 +1,6 @@ +--- +date: 2021-10-08 +--- + +# Emergency Mirrors Outage 10/8 +

Our mirror server is currently down for maintenance. ETA 60 mins.


Update: 1530 PST This incident has been resolved.

diff --git a/docs/news/posts/blogger-post-2021-10-08-2.md b/docs/news/posts/blogger-post-2021-10-08-2.md new file mode 100644 index 0000000..98e4332 --- /dev/null +++ b/docs/news/posts/blogger-post-2021-10-08-2.md @@ -0,0 +1,6 @@ +--- +date: 2021-10-08 +--- + +# Tanglu, Hyperbola decommissioning +

Update: Maintenance was completed during Friday's downtime. No service interruptions expected today. Tanglu and Hyperbola mirrors have been decommissioned.


We are planning to take the mirrors server down for disk array migration from 10am-6pm this Saturday. HTTP, HTTPS, FTP, and Rsync will be unavailable during the maintenance window. The server may come up before 6pm, but please expect degraded performance and/or service interruption throughout the maintenance window.

We are planning to decommission tanglu and tanglu-releases as the project has been defunct for over 2 years. We strongly recommend against the continued use of Tanglu, but if you are still using Tanglu, please change your source.list to another archive mirror.  As of Oct 7, 2021, http://ftp-stud.fht-esslingen.de/Mirrors/archive.tanglu.org/tanglu/ is available.

We are also planning to decommission Hyperbola. Our Hyperbola mirror has been out of date for quite a while now (almost 3 years!), but please change your configuration accordingly if you are using Hyperbola and still have us as an entry.

We apologize for the inconvenience.

OCF staff

diff --git a/docs/news/posts/blogger-post-2021-10-11-1.md b/docs/news/posts/blogger-post-2021-10-11-1.md new file mode 100644 index 0000000..0d05fc6 --- /dev/null +++ b/docs/news/posts/blogger-post-2021-10-11-1.md @@ -0,0 +1,6 @@ +--- +date: 2021-10-11 +--- + +# [Investigating] Mirrors disk status +

10/11 0000 PST: We noticed that one disk attribute was abnormal on multiple disks in the mirrors server. All SMART attributes are currently normal, so we currently do not expect any immediate service interruption as we investigate the issue. 

10/12: Scheduled maintenance at 1930 PST. ETA 15 mins.

We will update this post once we have more information on the situation.

diff --git a/docs/news/posts/blogger-post-2021-10-23-1.md b/docs/news/posts/blogger-post-2021-10-23-1.md new file mode 100644 index 0000000..8fe464d --- /dev/null +++ b/docs/news/posts/blogger-post-2021-10-23-1.md @@ -0,0 +1,6 @@ +--- +date: 2021-10-23 +--- + +# centos-debuginfo mirror decommissioning +

For years, we have been mirroring the centos-debuginfo project, which hosts the *-debuginfo packages for CentOS. However, our centos-debuginfo mirror is an unofficial one, and is not listed on the CentOS website. 

Our traffic statistics show that the mirror has no to negligible actual usage, with tens to hundreds of megabytes of data transferred daily (almost entirely by bots). 

The project currently occupies 2.7 TiB of disk space, which is the highest of all the projects we mirror, and yet no user is actually using it. As a result, we will decommission the centos-debuginfo mirror on Saturday, October 30, 2021 at around 1200 PDT. The project will continue to be synced in the meantime.

If you are currently using our centos-debuginfo mirror, or if you have any concerns or questions, please email us as soon as possible at help@ocf.berkeley.edu.

We expect to have 3 TiB of usable disk space on the mirrors server following the removal of centos-debuginfo, so if you have any suggestions for open source projects to mirror, please also email us at help@ocf.berkeley.edu :) We are currently discussing adding ubuntu-ports. 

rjz

OCF Staff

diff --git a/docs/news/posts/blogger-post-2021-10-24-1.md b/docs/news/posts/blogger-post-2021-10-24-1.md new file mode 100644 index 0000000..830b2fa --- /dev/null +++ b/docs/news/posts/blogger-post-2021-10-24-1.md @@ -0,0 +1,6 @@ +--- +date: 2021-10-24 +--- + +# HTTP/2 is now enabled for web hosting +

 HTTP/2 is enabled for https connections on the web hosting server (death) as of 1730 PDT, Oct. 23.


Please email us at help [at] ocf.berkeley.edu (or send a message in #rebuild) if you experience any issues.

diff --git a/docs/news/posts/blogger-post-2021-11-02-1.md b/docs/news/posts/blogger-post-2021-11-02-1.md new file mode 100644 index 0000000..b8e3ca9 --- /dev/null +++ b/docs/news/posts/blogger-post-2021-11-02-1.md @@ -0,0 +1,6 @@ +--- +date: 2021-11-02 +--- + +# [RESOLVED] Scheduled downtime on 11/13 +

 We are planning on having some scheduled downtime on Saturday 11/13 starting around 12:00pm PST. We expect to be done before 4pm. We will be rebooting our main hypervisor as well as some other machines. Our website, student group websites, and other OCF services will be unavailable for the duration of this downtime.

If you have any questions about the downtime please contact us.


Update 3:57PM 2021-11-13: We have experienced some issues updating our main production server, so we expect to take at least another hour. We apologize for the inconvenience.

Update 7:14PM 2021-11-13: We have been told to leave the build which hosts our servers by the Student Union staff, therefore we likely will not be able to bring things back up until Monday.

Update 4:49AM 2021-11-14: Most services are now up. We are working on bringing scurvy (staff VM, Berkeleytime etc.) up.

Update 5:28AM 2021-11-14: Scurvy is up, berkeleytime is up, hellfire is up, marking this as RESOLVED?

Update 5:40AM 2021-11-14: Marking this as RESOLVED

diff --git a/docs/news/posts/blogger-post-2021-11-14-1.md b/docs/news/posts/blogger-post-2021-11-14-1.md new file mode 100644 index 0000000..a4c8d28 --- /dev/null +++ b/docs/news/posts/blogger-post-2021-11-14-1.md @@ -0,0 +1,6 @@ +--- +date: 2021-11-14 +--- + +# Mirrors scrub in progress +

 Mirrors FS scrub in progress, performance may be impacted. ETA 48 hours.

diff --git a/docs/news/posts/blogger-post-2021-11-18-1.md b/docs/news/posts/blogger-post-2021-11-18-1.md new file mode 100644 index 0000000..58fbae9 --- /dev/null +++ b/docs/news/posts/blogger-post-2021-11-18-1.md @@ -0,0 +1,6 @@ +--- +date: 2021-11-18 +--- + +# Scheduled downtime on 11/20 +

Most OCF services will be down on Saturday, 11/20 due to a scheduled hypervisor OS upgrade. Updates will be posted on this page.

diff --git a/docs/news/posts/blogger-post-2021-12-20-1.md b/docs/news/posts/blogger-post-2021-12-20-1.md new file mode 100644 index 0000000..6294861 --- /dev/null +++ b/docs/news/posts/blogger-post-2021-12-20-1.md @@ -0,0 +1,6 @@ +--- +date: 2021-12-20 +--- + +# None + diff --git a/docs/news/posts/blogger-post-2021-12-20-2.md b/docs/news/posts/blogger-post-2021-12-20-2.md new file mode 100644 index 0000000..6294861 --- /dev/null +++ b/docs/news/posts/blogger-post-2021-12-20-2.md @@ -0,0 +1,6 @@ +--- +date: 2021-12-20 +--- + +# None + diff --git a/docs/news/posts/blogger-post-2022-01-21-1.md b/docs/news/posts/blogger-post-2022-01-21-1.md new file mode 100644 index 0000000..272a921 --- /dev/null +++ b/docs/news/posts/blogger-post-2022-01-21-1.md @@ -0,0 +1,6 @@ +--- +date: 2022-01-21 +--- + +# Planned server reboot on 1/22 +

We will be rebooting our servers on Saturday, 1/22 at 2:00 PM PST for kernel updates. We expect the downtime to be around 30 minutes.

diff --git a/docs/news/posts/blogger-post-2022-02-05-1.md b/docs/news/posts/blogger-post-2022-02-05-1.md new file mode 100644 index 0000000..cb8eadb --- /dev/null +++ b/docs/news/posts/blogger-post-2022-02-05-1.md @@ -0,0 +1,6 @@ +--- +date: 2022-02-05 +--- + +# Mirrors upgrade underway, service interruptions expected +

We are currently performing upgrades on the OCF mirrors server to accommodate more projects and improve the throughput. During this time, you might experience sporadic downtime and reduced performance. Please retry your downloads in a few seconds if you experience failures when downloading from the OCF mirrors.

We apologize for any inconvenience. We expect the upgrades to be completed by mid-February.

Later this semester, we will migrate the mirrors server to the campus datacenter to ease up the network load at the OCF lab*. The project is currently in the planning stages, so no definite dates are available. However, we will make a separate post on the status blog before we proceed with the migration. It is likely that the server's IP addresses will fall outside of the OCF subnet as a result of the migration.


Planned additions as of 2/6 2230PST (please note, this list is not final and may change without notice):

Fedora - initial sync underway

OpenSUSE - initial sync underway

Gentoo - up

Ubuntu Ports (releases) - up

Blender - up

Debian non-free images - initial sync underway

Xfce

CentOS Stream - initial sync underway

Rocky Linux - initial sync underway

Qubes OS - up


Feel free to email help@ocf.berkeley.edu if you have any suggestions :)


*: subject to change

diff --git a/docs/news/posts/blogger-post-2022-02-07-1.md b/docs/news/posts/blogger-post-2022-02-07-1.md new file mode 100644 index 0000000..6829326 --- /dev/null +++ b/docs/news/posts/blogger-post-2022-02-07-1.md @@ -0,0 +1,6 @@ +--- +date: 2022-02-07 +--- + +# None + diff --git a/docs/news/posts/blogger-post-2022-02-08-1.md b/docs/news/posts/blogger-post-2022-02-08-1.md new file mode 100644 index 0000000..d407213 --- /dev/null +++ b/docs/news/posts/blogger-post-2022-02-08-1.md @@ -0,0 +1,6 @@ +--- +date: 2022-02-08 +--- + +# Scheduled downtime on Saturday, 2/12 +

We will be upgrading the operating system on our web hosting (death), login (tsunami), and app hosting (vampires) servers this Saturday, 2/12/2022, beginning at 6:00 pm Pacific Time. All OCF-hosted websites and apps will be unavailable during the upgrade. We expect the downtime to last for about 4 hours.

We apologize for the inconvenience.

diff --git a/docs/news/posts/blogger-post-2022-02-14-1.md b/docs/news/posts/blogger-post-2022-02-14-1.md new file mode 100644 index 0000000..b34fc2e --- /dev/null +++ b/docs/news/posts/blogger-post-2022-02-14-1.md @@ -0,0 +1,6 @@ +--- +date: 2022-02-14 +--- + +# Mirrors maintenance tonight at 8 +

We will be installing additional RAM on the mirrors server tonight at 8pm. Expect 15 mins of downtime.



Update: ETA 2130. Apologies, but the process turned out to be more involved than expected.

Update 2: The mirrors are up as of 21:30.

diff --git a/docs/news/posts/blogger-post-2022-03-11-1.md b/docs/news/posts/blogger-post-2022-03-11-1.md new file mode 100644 index 0000000..49c863b --- /dev/null +++ b/docs/news/posts/blogger-post-2022-03-11-1.md @@ -0,0 +1,6 @@ +--- +date: 2022-03-11 +--- + +# [Complete, see notes] Mirrors downtime 3/11/2022 - migration to campus DC +

 migration to campus DC underway, expect downtime for ~2 hours


Update 3/12/2022: Migration completed by 14:00 yesterday. Currently our mirrors are IPv4 only, IPv6 ETA 2 weeks. Some Kali packages may not download correctly due to campus firewall settings, so if you experience any broken Kali downloads, please switch to the https version of our mirror. We are in contact with the university to get the issue sorted out as soon as possible.

diff --git a/docs/news/posts/blogger-post-2022-03-13-1.md b/docs/news/posts/blogger-post-2022-03-13-1.md new file mode 100644 index 0000000..2f1376a --- /dev/null +++ b/docs/news/posts/blogger-post-2022-03-13-1.md @@ -0,0 +1,6 @@ +--- +date: 2022-03-13 +--- + +# Emergency maintenance: hypervisors reboot today +

We will be rebooting the hypervisors this afternoon at about 2 pm to apply the updated kernel that fixes several high-severity vulnerabilities. The downtime will be around 15-30 minutes, and all websites hosted by the OCF as well as the login servers, Matrix server, Slack bridge, mail server, and the IRC server will be unavailable during the reboot.

diff --git a/docs/news/posts/blogger-post-2022-03-15-1.md b/docs/news/posts/blogger-post-2022-03-15-1.md new file mode 100644 index 0000000..2b17227 --- /dev/null +++ b/docs/news/posts/blogger-post-2022-03-15-1.md @@ -0,0 +1,6 @@ +--- +date: 2022-03-15 +--- + +# Emergency mirrors server maintenance today +

 The OCF mirrors server will be down for about an hour staring 1300 PDT today due to a required move within the rack and IPv6 provisioning. We apologize for the inconvenience.


diff --git a/docs/news/posts/blogger-post-2022-04-03-1.md b/docs/news/posts/blogger-post-2022-04-03-1.md new file mode 100644 index 0000000..3e54028 --- /dev/null +++ b/docs/news/posts/blogger-post-2022-04-03-1.md @@ -0,0 +1,6 @@ +--- +date: 2022-04-03 +--- + +# [RESOLVED] Raspbian out of sync +

 We noticed that our Raspbian mirror is out of sync, likely due to the IP address change. 

10:16 PM - We have temporarily switched our upstream for the Raspbian mirror, and it's currently syncing. Emailed Raspbian. - rjz

5/13 - Resolved.

diff --git a/docs/news/posts/blogger-post-2022-05-13-1.md b/docs/news/posts/blogger-post-2022-05-13-1.md new file mode 100644 index 0000000..39649f0 --- /dev/null +++ b/docs/news/posts/blogger-post-2022-05-13-1.md @@ -0,0 +1,6 @@ +--- +date: 2022-05-13 +--- + +# Fedora out of sync +

 I noticed that our Fedora mirror was out of sync due to a slow upstream mirror. Working on it.


Update: Switched to another upstream mirror. Sync speeds are higher now.

diff --git a/docs/news/posts/blogger-post-2022-08-27-1.md b/docs/news/posts/blogger-post-2022-08-27-1.md new file mode 100644 index 0000000..98d66aa --- /dev/null +++ b/docs/news/posts/blogger-post-2022-08-27-1.md @@ -0,0 +1,6 @@ +--- +date: 2022-08-27 +--- + +# Multiple mirrors out of date +

 We noticed that multiple OCF mirror archives were out of date due to exhausted boot disk space on the mirrors server.

Affected archives include (this is not an exhaustive list):

Debian

Debian-security

Puppetlabs

Ubuntu

Ubuntu-ports

Kali


The last update of the affected archives was on Aug 13. Sync is running but might take a while.

We sincerely apologize for this. We will take steps to ensure that system monitoring alerts are handled in a timely manner.



diff --git a/docs/news/posts/blogger-post-2022-09-11-1.md b/docs/news/posts/blogger-post-2022-09-11-1.md new file mode 100644 index 0000000..d08ebb1 --- /dev/null +++ b/docs/news/posts/blogger-post-2022-09-11-1.md @@ -0,0 +1,6 @@ +--- +date: 2022-09-11 +--- + +# Urgent Maintence Downtime +

tl;dr; The entire OCF will be offline for at least 24 hours.

We have discovered that several of the drives in our main storage array are experiencing a firmware bug that may lead to data loss. We are focusing on minimizing the impact of this issue and as a result we have decided to take the entire OCF offline for at least 24 hours until we can get replacement drives. We will be posting updates about the downtime to this blog, so keep an eye out.


2022-09-11 3:08PM: OCF goes offline

diff --git a/docs/news/posts/blogger-post-2023-01-18-1.md b/docs/news/posts/blogger-post-2023-01-18-1.md new file mode 100644 index 0000000..e161e7a --- /dev/null +++ b/docs/news/posts/blogger-post-2023-01-18-1.md @@ -0,0 +1,6 @@ +--- +date: 2023-01-18 +--- + +# Mirrors maintenance, 1/18 23:00 PST +

We'll be running some maintenance tasks on the mirrors server (https://mirrors.ocf.berkeley.edu) at 23:00 PST today, 1/18.

The projected maintenance window is 20 minutes.

diff --git a/docs/news/posts/blogger-post-2024-02-09-1.md b/docs/news/posts/blogger-post-2024-02-09-1.md new file mode 100644 index 0000000..8c24d75 --- /dev/null +++ b/docs/news/posts/blogger-post-2024-02-09-1.md @@ -0,0 +1,6 @@ +--- +date: 2024-02-09 +--- + +# OCF Mirrors Down, Feb 8 ~11AM - Feb 10 9PM +

Update (2/10 9PM PST) - OCF Mirrors Service is up again, and we are currently syncing from upstreams. OCF Internal APT repos remain unavailable at this moment.

 The OCF Mirrors Service went down due to a campus datacenter power outage. When power was resumed, the server encountered a disk error on the boot disk.


ETA of recovery currently unknown as a full OS rebuild seems to be needed.


~rjz

diff --git a/docs/news/posts/blogger-post-2024-08-11-1.md b/docs/news/posts/blogger-post-2024-08-11-1.md new file mode 100644 index 0000000..c881c02 --- /dev/null +++ b/docs/news/posts/blogger-post-2024-08-11-1.md @@ -0,0 +1,6 @@ +--- +date: 2024-08-11 +--- + +# Planned Outage: All OCF Services, August 14 +

 All OCF services will be unavailable on Wednesday, August 14, due to a planned electrical shutdown in the MLK Student Union building.

While the electrical shutdown does not affect the campus data center, the availability of the OCF Software Mirror service. The OCF HPC service will be unavailable during the outage.

diff --git a/docs/news/posts/blogger-post-2024-08-11-2.md b/docs/news/posts/blogger-post-2024-08-11-2.md new file mode 100644 index 0000000..cd3c547 --- /dev/null +++ b/docs/news/posts/blogger-post-2024-08-11-2.md @@ -0,0 +1,6 @@ +--- +date: 2024-08-11 +--- + +# Planned Outage: OCF Mirrors, Aug. 11 +

 Due to a scheduled power outage at the UC Berkeley Earl Warren Data Center, OCF Mirrors (https://mirrors.ocf.berkeley.edu) will be unavailable starting 4 AM PDT, Sunday Aug. 11.

The server is expected to be online by 7 PM PDT, Sunday Aug. 11.