Often when I read ML papers the authors compare their results against a benchmark (e.g. using RMSE, accuracy, …) and say “our results improved with our new method by X%”. Nobody makes a significance test if the new method Y outperforms benchmark Z. Is there a reason why? Especially when you break your results down e.g. to the anaylsis of certain classes in object classification this seems important for me. Or do I overlook something?

  • AwarenessPlayful7384@alien.topB
    link
    fedilink
    English
    arrow-up
    1
    ·
    1 year ago

    Cuz each experiment is too expensive so sometimes it just doesn’t make sense to do that. Imagine training a large model on a huge dataset several times in order to have a numerical mean and variance that dont mean much.