Talk:Hardware deployed May 9, 2005

This is currently a provisional order list (april 2005), still being discussed, not a list of what has been ordered.

Database notes 5 June 2005: number of disks here may (hopefully) make DB servers CPU-limited (in which case plan is quad CPU or dual dual-core CPU box to see where the balance point is). MediaWiki 1.5's changes to cur and old may also do that, need to evaluate whether it does so will know if can use less or larger/slower seeking drives. Need to evaluate larger capacity fast SCSI drives now becoming available. DB servers now not being used for search have much more RAM available to be allocated to InnoDB caching, so that should keep them able to handle things for a fair bit of growth beyond April - search was really hurting. Also need to actually order those non-P4 Apaches so we can test and benchmark them. Jamesday 22:06, 5 Jun 2005 (UTC)

Rack Power Distribution Units (PDU), one per rack. Two power outage incidents have been related to circuit overloads. The PDUs provide monitoring of total power draw, reducing the chance of futher outages for this reason. 3 of APC 7832 for $365 each.

Redundant power module for main Cisco switch, PWR675-RPS-N1 or equivalent. This is a key single point of failure for the site and needs protection.

For recovery from major failures, additional backup and log storage on all existing database servers. For master servers, the logs here are the long term binary logs space for which has twice filled the costly and small SCSI drives. The larger SATA drive and a log moving script is expected to reduce the chance of repeats of unexpected master database server switches for this reason. One 400GB SATA drive for five servers, to go in the internal drive bay. This will allow each to have its own local backup with storage for logs holding all changes since the last backup, allowing roll-forward restoration of each server independent of the others. The drive will also improve general performance by being used for temporary table storage during queries. The 3 existing SCSI servers do not have a SATA connector so will need a PATA to SATA adaptor for the drive (all SATA drives to keep all our drives SATA for simplicity). Need to check with SM whether the two SATA boxes have the motherboard SATA module or also need PATA to SATA option. Cable length needed is about 24" (about width of a rackmount case).

For some possible donated equipment and for sites with limited rack slots, also because it appears that dual Xeons will be more cost-effective than a pair of single P4s, want to evaluate one dual Xeon and one dual Opteron 1U box (for reference, Liveournal uses dual Xeons and one major potential hardware donor is also specifying them). Need to test them to see if they deliver and to evaluate in our load balancing setup.

Our current database server set is being adjusted to have 400GB of usable space on all servers. The current 4GB sets are showing some strain for en and de with the large record size of cur, commonly used to get just the 200MB or so of titles but needing 3.5GB to cache because it includes the whole of the article text. Mediawiki 1.5 is expected to address this issue and substantially improve performance.

For Suda, a pair of 140GB SCSI drives comparable to its current drives. These will take it to 400GB RAID 10 capacity as a master-capable server. Being considered only: 8GB of additional RAM so it can more easily handle master server load. May be better to do this in another new server.

New primary master database server, with 16GB of RAM, dual Opterons and six 140GB 15,000 RPM SCSI drives in RAID 0, delivering 420GB of usable space. Caching SCSI controller. 12*Fujitsu 73/74GB 15K RPM SCSI drives in the external case. 400GB SATA drive in internal bay and motherboard SATA controller added. This will partly evaluate the usefullness of 16GB of RAM. Once this is proved in service, Ariel is planned to be switched to RAID 0 to give it 400GB of usable disk space.

Silicon Mechanics SM-2280S with:

• Dual Opteron 244 CPU
• 16GB (8*2GB) RAM
• internal hot-swap SCSI drives: 6*Fujitsu 73 or 74GB 15K RPM SCSI (capacity is whatever the drives are - it's about 73 or 74GB)
• external 2U 12 drive SCSI case Ken and I discussed (not the nStor or Adaptec), about $1,000
• 12*Fujitsu 73/74GB 15K RPM SCSI drives in the external case
• 1*400GB SATA drive in the internal (not hot swap) bay and the motherboard SATA controller option it needs
• LSI MegaRAID SCSI 320-4x - 4 channel U320 RAID with BBU. Comes with 128MB of cache, ask for higher cache option prices
• Drive configuration: the SCSI drives with two hot spare and 16 in RAID 10 (mix across different channels for each mirrored pair, of course). Drive write caching off, controller write caching on, read caching off. And test that it really survives power out. Use just two of the external connectors - want option of adding another external JBOD later.

En/ja Wikipedia search/general database slave. For en Wikipedia with Mediawiki 1.4, 4GB is no longer sufficient to cache the amount of data in cur and the search indexes. To provide additional search capability and to evaluate performance with 8GB, a dual Opteron box identical to Holbach but with 8GB of RAM instead of 4GB. SATA has proved generally comparable to SCSI except under high replication load and the en/ja set of database servers is expected to see comparatively high repliation write loads, so SCSI drives are better than SATA for this set. No need for RAID 10 for this slave-only system. Should evaluate 140GB 10K SCSI drives to see if this might be best purchased with 800GB usable space to be part of the terrabyte-usable capacity target for our next generation of servers. 400GB SATA drive in internal bay and motherboard SATA controller added. The Seagate 400GB drive for the internal bay is SM part 12284 $409 and onboard SATA controller for the motherboard is part number 11337 for $42. Have to ask for a specific quote for this internal bay - can't configure it via the site.

De Wikipedia is now an uncomfortably high load for Webster, again for the temporary RAM capacity reasons with Mediawiki 1.4. Same specificaton as en/ja.

for en/ja and de/rest, two of these systems:

Silicon Mechanics SM-2280S with:

• Dual Opteron 244 CPU
• 8GB (4*2GB) RAM
• internal hot-swap SCSI drives: 6* 73 or 74GB 10K RPM SCSI (capacity is whatever the drives are - it's about 73 or 74GB)
• external 2U 12 drive SCSI case Ken and I discussed (not the nStor or Adaptec), about $1,000
• 12* 73/74GB 10K RPM SCSI drives in the external case
• 1*400GB SATA drive in the internal (not hot swap) bay and the motherboard SATA controller option it needs
• LSI MegaRAID SCSI 320-4x - 4 channel U320 RAID with BBU. Comes with 128MB of cache, ask for higher cache option prices
• Drive configuration: the SCSI drives with two hot spare and 16 in RAID 10 (mix across different channels for each mirrored pair, of course). Drive write caching off, controller write caching on, read caching off. And test that it really survives power out. Use just two of the external connectors - want option of adding another external JBOD later.

Albert is disk speed limited and site performance has been suffering because of it. As a possibly temporary expedient to address this, a new box with dual Opterons, 6x400GB SATA drives in RAID 10 with a write caching SATA controller and 8GB of system RAM for caching. If changes to file handling make this system unnecessary it can be switched to database service. Consider 400GB SATA drive in internal bay for temporary file storage while preparing database and image dumps, off the main drives to reduce load on them using non-redundant scratch storage.

• How is the caching of image data working? Do we still need this? Will use it for DB anyway if not... Jamesday 09:02, 6 Apr 2005 (UTC)

Search. 4GB is now too little to hold the search indexes and cur for most projects, which is what is necessary for it to work well, so more RAM for all existing database servers will help. Can wait and see how much difference it makes for the two new 8GB slaves. After that expect 4GB more for Bacon, Holbach, Webster (and the 8GB for Suda). A software solution (taking search off database servers) is also being worked on so this may not be necessary.

Power failure note

Both disk controller vendors have now addressed the write caching problems which conributed to our (and LJ) power loss trouble. The SATA vendor now has a new driver and the SCSI vendor a new utility to turn off hard drive write caching, with more work planned. For this reason, no switch of controller vendor is planned for this batch of database servers.

Tentative Order

 
Item                                          Cost:

1 Dual Xeon 1U box                                 ?
1 Dual Opteron 1U box                              ?
1 Dual Opteron,16GB RAM, 6x 140GB 15k SCSI HDDs,
  Caching SCSI controller, 1x 400GB sata HDD 
  (new master)                                     $ =)
2 Dual Opteron, 6x73GB RAID 0 or 140GB SCSI HDDs, 
  1 400GB SATA in internal bay, 8GB RAM
  (en/ja and de)                                   ?
1 Dual Opteron, 8GB RAM, 6 RAID 10 Seagate 400GB
 (7200.8) SATA HDD and SATA 400GB in internal bay 
 (albert)                                          ?
3 Power Distribution Units APC 7832                $1095
1 Redundant Power Module PWR675-RPS-N1             ?
5 400GB SATA HDDs                                  ?
2 140GB 10K RPM SCSI HDDs                          ?
3 PATA to SATA adapters                            ?
1 8GB RAM (suda possibility)                       ?

  

                                             Total:?
                                         Available:~96,000