DevOps Research Model to Accelerate & Measure Progress in Machine Vision

[josh]

Q: This notion of “testing” has some similarities & some differences compared to concepts like unit testing & regression testing in software development. What translates & what doesn’t?

A: Testing the overall pipeline performance for accuracy is like performance testing in software development. Some types of performance which should be considered “easy” & not get broken by any change can be translated to unit tests. Of course you don’t want to the software to misfunction simply as software & that element is the same. The overall battery of tests will be more computationally expensive to run than product performance tests for most software so more attention should be given to early squelching of unpromising candidates – they don’t need anything like a complete run to be rejected as a performance regression.

[josh]

Q: If there are N candidate models for a similar stage of elaboration & they all share a similar level of overall performance, what is the correct strategy for pruning?

A: It depends on the following factors

1. Are they arriving at similar performance by giving the same answers in individual instances or are they really giving different answers?
2. Are variations in answers at this stage impacting output performance?
3. Can we find an improved decision mechanism to choose among them according to case factors?
4. Do the models represent different concepts that are interesting to scientist engineers? In general, the current main performance branch will be affected by synergy factors that veer away from scientific clarity & a displaced algorithm is not a sign of a displaced scientific conception.

Global optimization from genetic algorithms/simulated annealing may make use of algorithmic variants, subspace methods may combine them, & statistical decision theory can inform about the relative solidity of preferences for 1 scheme over another at present.

Try to have some smart human managers & a bulletin/newsletter to help with decisions & giving informative synopsis of developments & focus areas.

[josh]

There are application-based reasons to support multiple pipeline profiles. These include variations in the available types of input & the required performance tasks. DevOps should support selecting particular profiles for testing, etc. The existence of an overarching intergrative framework for continuing research development & comprehensive performance evaluation should ultimately help all application areas.

[josh]

Food for thought: Say that you want to investigate models that combine bottom-up & top-down information through sequentially adaptive decisions about different image regions. What has to be true of various software routines to enable a sort of blackboard architecture that calls on different routines to elaborate only parts of the image in a data dependent order? Does this researcher have to reprogram a lot of other software to try it out in this framework or is there some valid way to simply use the existing parts??

I believe, unfortunately, that existing algorithms do not support this. Not because it is silly, but simply because it wasn’t easy or clean for them to do. The existence of a dev ops framework should at least support someone starting down that path to see if it is helpful.

[josh]

If all software routines in your eval set supported input that included “not yet defined” at this location, then you could optimize by doing a meta-analysis on training data to see where mistakes were least common & then using algorithmic learning of those features to insist that decisions occur first at those image coordinates.

[josh]

Topic: Assuming we want to decompose Machine Vision Research & NLP research into different units, how can we test Machine Vision for symbolic understanding of scenes?

One idea: Make a separate project that generates VR scenes from linguistic descriptions. Of course each output gets tagged with the program version so we can update/improve that over time. A human can judge the overall performance or individual efforts. For each VR we can use graphics to create images of the whole or part of the scene. Machine vision programs can then be run on training/test that shows an understanding of which part types were found in the images and which were not, without referencing word understanding per see.

[josh]

One might worry that simple VR doesn’t have the confounding conditions of real world images. Scientifically though, it’s interesting to build a bridge from both directions. We can approach a tough image from harder & harder VR constructions with more simulated imaging artifacts and different degrees of shape similarity to what a human sees in the “difficult” real image case.

[pers_d7pyzaKeymaster]

The consensus view in the graphics community is that the general framework of ray tracing can be used to theoretically describe most of the extra complexity in real images. However, ray tracing as implemented in graphics practice for VR/CGI typically simplifies the image formation process by assuming constant or simple gradient refractive indexes for the media that the rays traverse. Obtaining greater levels of realism is possible by increasing the complexity of image formation model & the ray tracing calculations arising from that. It’s a interesting area for GT because it’s relevant to this scientific/engineering project, we don’t have a need for real time rendering, and the most effective techniques may become part of next generation CGI.

[Author]

Posts