User talk:Milimetric (WMF)/Work log/2015-06-26

Active discussions

Friday, June 26, 2015Edit

Event Logging load testEdit


  • using analytics1004.eqiad.wmnet, a beefy if not trusty Cisco box
  • 31492 events, 4 schemas, split 3/3/3/1, generated by the wonderful
  • format: seqId[space]?[qson event]


  • 4 runs average: 17.6478 seconds, so ~1784 events per second
  • command: time cat /home/milimetric/load.test.seq.30k | python ./eventlogging-processor "%{seqId}d %q" stdin:// file:///home/milimetric/out.load.30k


  • NOTE: mysql server was installed on the same host as the eventlogging-consumer
  • NOTE: tried this with four different batch sizes (400, 1000, 4000, 8000) and it didn't seem to make a substantive difference. Therefore I just used the default configured batch size of 400. This indicates that most likely the performance bottleneck is in preparing and organizing the events for insertion and not the insertion itself.
  • 4 runs average: 11.8945 seconds, so ~2647 events per second
  • command: time cat ~/out.load.30k | python eventlogging-consumer stdin:// mysql://root:root@
Return to the user page of "Milimetric (WMF)/Work log/2015-06-26".