The article Why Machine Learning Doesn’t Work Well for Some Problems? (Shahab , 2017) describes the effect of Emergence as a barrier for predictive inference.
Emergence is the phenomenon of completely new behavior arising (emerging) from interactions of elementary entities, such as life emerging from biochemistry and collective intelligence emerging from social animals.
In general, effects of emergence cannot be inferred through a priori analysis of a system (or its elementary entities). While weak emergence can be understood still by observing or simulating the system, emergent qualities from strong emergence cannot be simulated with current systems.
Sheikh-Bahei suggests interpreting emergence (in a predictive context) as an additional dimension, called the E-Dimension, where moving along that dimension results in new qualities emerging. Crossing E-Dimensions during inferrence leads to reduced predictive power as emergent qualities cannot be necessarily described as a function of the observed features alone. The more E-Dimensions are crossed during inferrence, the lower the prediction success will be, regardless of the amount of feature noise. Current-generation algorithms do not handle this kind of problem well and further research is required in this area.
Hypothetical example of the E-Dimension concept: Emergence phenomena can be considered as a barrier for making predictive inferences. The further away the target is from features along this dimension, the less information the features provide about the target. The figure shows an example of predicting organism level properties (target) using molecular and physicochemical properties (feature space). (Shahab , 2017)
| Example ML problem | Feature space | Feature noise | Emergence barrier | Prediction success |
|---|---|---|---|---|
| Character recognition | handwritten character images | high | none | high |
| Speech recognition | sound waves | high | none | high |
| Weather predictions | climate sensor data | high | weak | high |
| Recommendation system | historic preferences, likes, etc. | low | weak | moderate |
| Ad-click prediction | historic click behavior | low | weak | moderate |
| Device failure prediction | sensor data | high | weak | moderate |
| Healthcare outcome predicitons | patient data, vital signs, behavior, etc. | high | strong | low |
| Melting/boiling point prediction | molecular/atomic structure | low | strong | low |
| Stock prediction | historic stock value, news articles, etc. | low | strong | low |