It is usually essential to make sure that the benchmark surroundings is much like the enterprise manufacturing surroundings, he stated, and to doc areas the place community, compute, storage, inputs, outputs, and contextual augmentation of the benchmark surroundings differ from the manufacturing surroundings.
Additional, make it possible for the mannequin examined matches the mannequin that’s obtainable for preview or for manufacturing, Park suggested. It is not uncommon for fashions to be optimized for a benchmark, with out revealing deep element into the associated fee or time required for the coaching, augmentation, or tuning going into that optimization.
Finally, “companies searching for to conduct a aggressive analysis of AI fashions can use benchmarks as a place to begin, however really want to state of affairs take a look at in their very own company or cloud environments if they need an correct understanding of how a mannequin may match for them,” Park emphasised.
