Q: This notion of “testing” has some similarities & some differences compared to concepts like unit testing & regression testing in software development. What translates & what doesn’t?

A: Testing the overall pipeline performance for accuracy is like performance testing in software development. Some types of performance which should be considered “easy” & not get broken by any change can be translated to unit tests. Of course you don’t want to the software to misfunction simply as software & that element is the same. The overall battery of tests will be more computationally expensive to run than product performance tests for most software so more attention should be given to early squelching of unpromising candidates – they don’t need anything like a complete run to be rejected as a performance regression.

Q: If there are N candidate models for a similar stage of elaboration & they all share a similar level of overall performance, what is the correct strategy for pruning?

A: It depends on the following factors

1. Are they arriving at similar performance by giving the same answers in individual instances or are they really giving different answers?
2. Are variations in answers at this stage impacting output performance?
3. Can we find an improved decision mechanism to choose among them according to case factors?
4. Do the models represent different concepts that are interesting to scientist engineers? In general, the current main performance branch will be affected by synergy factors that veer away from scientific clarity & a displaced algorithm is not a sign of a displaced scientific conception.

Global optimization from genetic algorithms/simulated annealing may make use of algorithmic variants, subspace methods may combine them, & statistical decision theory can inform about the relative solidity of preferences for 1 scheme over another at present.

Try to have some smart human managers & a bulletin/newsletter to help with decisions & giving informative synopsis of developments & focus areas.

There are application-based reasons to support multiple pipeline profiles. These include variations in the available types of input & the required performance tasks. DevOps should support selecting particular profiles for testing, etc. The existence of an overarching intergrative framework for continuing research development & comprehensive performance evaluation should ultimately help all application areas.

Food for thought: Say that you want to investigate models that combine bottom-up & top-down information through sequentially adaptive decisions about different image regions. What has to be true of various software routines to enable a sort of blackboard architecture that calls on different routines to elaborate only parts of the image in a data dependent order? Does this researcher have to reprogram a lot of other software to try it out in this framework or is there some valid way to simply use the existing parts??

I believe, unfortunately, that existing algorithms do not support this. Not because it is silly, but simply because it wasn’t easy or clean for them to do. The existence of a dev ops framework should at least support someone starting down that path to see if it is helpful.

Topic: Assuming we want to decompose Machine Vision Research & NLP research into different units, how can we test Machine Vision for symbolic understanding of scenes?

One idea: Make a separate project that generates VR scenes from linguistic descriptions. Of course each output gets tagged with the program version so we can update/improve that over time. A human can judge the overall performance or individual efforts. For each VR we can use graphics to create images of the whole or part of the scene. Machine vision programs can then be run on training/test that shows an understanding of which part types were found in the images and which were not, without referencing word understanding per see.