I read Cathy O'Neil's Weapons of Math Destruction a few weeks ago and I continue to mull it over, and I want to spend some time expanding on one small part of the book. In WMD Ms. O'Neil talks a fair bit about how models can lead to terrible outcomes due to a fusion of their particular blindspots and perverse incentives. One thing I would like to expand upon is the how these blindspots can develop naturally and be obscured by the naive performance metrics that one typically uses to decide how well the model is functioning.
This is one of those questions I could probably answer with research, but I'm lazy so I am going to do simulations. Anyways, it came up in my life to check a data-set to see if the values are normally distributed. There are a couple of ways of doing this (I lean towards doing a KS test) but one that was recommended was to do a $\chi^2$ test. Of course the $\chi^2$ test typically requires the data to be in discrete bins, and this got me thinking: surely the test itself is highly dependent upon the bin size I choose so, presumably, I could fiddle with that variable to get whatever answer I wanted. Presumably.
Recently I was talking, on the internet no less, with someone who was trying to take some data presumably drawn at random and find the parameters of the corresponding pdf (some weird Weibull something-er-other for doing risk modelling but that's beside the point for what follows). In simple cases the common man would look up the maximum likelihood estimators for the parameters of the given pdf, or maybe numerically find the parameters that minimize the Kolmogorov-Smirnov statistic or something. This guy's plan was to do a non-linear least squares fit of the pdf.