Minutes of the SIPB Meeting of 2008-09-15 The meeting was called to order at 19:30 by price. In attendance were Voting members: jesstess, price, presbrey, quentin, nelhage, kaduk, ternus, broder, mitchb, jhawk | golem Associate members: asedeno Prospectives: dbankman, biyeun, tvald, rishig, jhamrick, sjlevine | xavid, pweaver Guests: gwen Officer Reports: price: How's does our budget look? kaduk: We currently are allocated $19,800 per year from OIS, but there are a large number of fees that we have very little control over. This includes about $7,000 for Athena soft/hardware fees for office machines given to us for free under a somewhat standard program, $65 per month for TSM backups of penguin-lust, charon, and sipb-afs (adding to $2,500 per year). This year to date, we've spent about $750 on other things, so there are just under $10,000 in discretionary funds for the remainder of FY09. Last year, we spent ~$1,500 on food for hackathons, etc., and office expenses (which are variable, so I haven't enumerated them). price: So about $10,000 is not firmly allocated, but we expect to spend about $1,500 for hackathons, etc., and maybe $1,000 for office related stuff, thus there's about $6,000-7,000 available for capital purchases this year? kaduk: Right. SIPB Projects Report: price: We had a documentathon yesterday; many people came. nelhage set up a wiki on which many people placed their documents. nelhage: The wiki to hold documentation is at http://sipb.mit.edu/doc jhamrick produced an introduction to zephyr for people who've never used it before (it talks about barnowl, as well as other clients that it generally discourages people from using in this day and age); there was some barnowl documentation by asedeno; quentin documented some scripts internals that only he understands. And we successfully held a day-shifted hackathon for the first time in a while (we started at 1, and people were there getting work done then). price: I'm happy to say that we'll be having Cluedumps again this year. They're a series of weekly or biweekly informal technical talks. The organizers this year are y_z and sunw. Look forward to hearing specifics from them on what will be cluedumped and when. You can talk to them, or if they're not here, me. kaduk: There will be an Ask SIPB column in this week's Tech, which is currently undergoing editing. It's a consolidation of the usual first two columns we run during Orientation week. We'll perhaps continue to run columns weekly this term. presbrey: We want to allocate money for SQL to buy RAM for the servers. We want to cache more data to avoid the read penalty from the slower disks. The proposal is to buy 4GB of RAM per server (2 servers, so 8GB total). $400 is the grand total for this. quentin: We chose Kingston RAM, which is cheaper than other options. presbrey: There is limited bandwidth for IO on the SQL servers, due to the single RAID-1 channel available and the amount of cache for it. The best long-term scenario is to increase the number of channels to disk and increase the RAM. There is limited money to buy new servers, so we want to buy RAM; this seems like the best way to improve the situation without wasting money. jhawk: How confident are you in your analysis of the bottleneck and the efficacy of the proposed solution? presbrey: Very. There appears to be a single user causing most of the load. broder: Does eliminating this user eliminate your need? presbrey: We've asked them to explain their usage better. Even without that user load, there's a significant amount of IOwait when that user is not using SQL. broder: This is a very large purchase sprung on SIPB without much time to consider. This is a smaller request than was previously mentioned [via e-mail], but $400 is still significant when we have $7,000 to get through the next 8 months. quentin: I don't want this wankfest to last another 6 months. jhawk: There should have been more notice, but a couple hundred dollars seems like a reasonable amount of money to spend on RAM to try upgrading one server and see how much of the desired result it has. I move to allocate $200 for RAM for one server. kaduk: I second. presbrey: I think you should vote yes for the RAM. We run Nagios to monitor loads, usage, etc. When things get too high, it e-mails us. We've tried to improve the load situation by turning off journaling, moving logs to some nefarious place, etc. <8-2-2 $200 to the SQL maintainers to upgrade RAM in one of their servers and evaluate the results before considering the other server> price: I think the lesson to take home here is that if you want to spend a lot of the Board's money, you have to give more than half a day to discuss details of it. mitchb: I do think that there definitely needed to be way more notice about the proposal. However, that aside, I think we have a more general problem. It shouldn't be the case that when one of our largest services, like SQL, is reaching a resource limit and wants to perform an obvious, simple, and not incredibly expensive upgrade like adding RAM, it requires great worry about whether we can fit it into our budget. It should almost be a no-op for us to say to SQL "of course you can have more RAM." I think we need to fix our funding situation. ternus: ==mitchb broder: I agree that we should address the funding problem. But I've spent much of the day questioning the SQL maintainers, and don't claim to understand enough of the details, though it seems like many of the answers they've given have been fairly handwavy. price: I agree with broder; I've heard much certainty from the SQL maintainers that the exact proposed (in private) plan was the right way to go. quentin: Is there a way to be sure that doubling the RAM in the service will have a particular quantity of impact on the performance? jhawk: Yes, but let's have this discussion offline. kaduk: Expanding on mitchb's comment, our budget this year is smaller than last year's, and last year we went and made some large purchases because we had a bit of a surplus. price: Our effective discretionary money is the same as in previous years, no? kaduk: There's some question about the ordering of various budget shrinkages occurring. price: But the effective budgets are the same this year as previously, right? kaduk: Our FY09 budget is the same as our FY08 one was at the *end* of FY08. jhawk: Because the FY08 budget changed out from under us. kaduk: Right. price: But we don't really have a different amount of discretionary money now than 2 years ago; we thought we had more last year. mitchb: I think the best way I can articulate the problem that we seem to be having now is that the server budget, which is where this type of expense would have gone in the past, has disappeared. That was in the last couple of years, right? price: It was two to three years ago, I think. mitchb: So, it's fantastic that Michael Gettes and Wilson D'Souza and company are providing huge amounts of funding for major projects, but it seems like we need to have a large funding proposal for a big project in order to receive that support, and that we're not really welcome to approach this process for routine purchases like RAM upgrades, and so they're now falling under the office budget, which isn't large enough to sustain them. Is there a way for us to begin trying to address this difficulty? kaduk: I'd be happy to correspond with them about the issue of liquid funding for maintenance of current services. price: I'm not sure we should go into more discussion of this right now, but we should discuss it. quentin: Do we want to say anything about XVM progress? price: XVM discovered that the 2 things EqualLogic suggested might fix our RAID problems did not fix them. jesstess: IAP is sooner than you think. Office Report: price: All of our laptop lands are full; there are people on the couch. jhawk: We lost a laptop land ethernet cable due to a crimp failure. kaduk: The monitor on opus is no longer functioning in parallel with the computer opus. This happened during the hackathon when I was out of the office. jhawk: I didn't notice it was a *DELL* monitor. I'll deal with it! [laughter] Other: quentin: Yesterday at the hackathon, we discovered that we didn't have enough chairs with arms. I just wanted to note that. broder: We discovered today that we don't have enough money to buy chairs with arms. mitchb: Maybe we have enough money to buy just the arms. nelhage: I'd just like to note that *quentin* found that we didn't have enough chairs with arms. price: We have a bound copy of the 6.828 xv6 sources. nelhage: The office copies of the IA32 manuals proved invaluable in demonstrating an error in the 6.828 website. dbankman: I notice that you don't have "Core Python" on your bookshelf, which is a python essential. I move to allocate money for it; it's listed as $34.64 on Amazon. jhawk: I second; $40.00. broder: geofft has an outstanding allocation for a number of books. Is this among them? [confirmation that it is not among geofft's allocated books] broder: White ballot? nelhage: There was a reporter from InfoWeek on campus today to learn about the "digital lifestyle" at MIT. She spoke with me and quentin about what SIPB's doing to improve it. nelhage: I was wearing this shirt [points to his "Shut the fuck up and write some code" Best Practical shirt], because it didn't occur to me this morning that it might be an unfortunate choice. kaduk: Part of the "digital lifestyle" is using zephyr during class to make both informative and snarky comments, and this has been a tradition for over 20 years. Other Other: '() The meeting was adjourned at 20:06. Minutes taken and submitted by mitchb.