Oct 10, 2003
-------------

- More on learning
	- learning requires bias
	- bias-variance dilemma holds
	- even within a bias, many solutions exist
		=> Occam's razor
	- bias can be too limiting

- Learning tasks
	- regression 
	- classification

- Learning methodology
	- use training set for learning
	- evaluate learning on test set

- Why is bias important?
	- motivation from #examples needed for
	  learning

- Example: learning boolean functions of n boolean variables
        - possibilities: |H| = 2^(2^n)
	- for two variables, leads to 16

- With every additional data point (example)
        - can rule out half of this

- Profile of narrowing down |H| with every example
        - but this is simple memorization

- Need generalization for good learning
        - what does the profile look like in this case?

- Lets adopt bias as conjunctions (n=2 case)
	- 10 possible hypotheses
		- 1
		- 0
		- a
		- not a
		- b
		- not b
		- not a and b 
		- not a and not b 
		- a and b 
		- a and not b 

- Consider the example data given by f=b
	- will need all four examples
	  to learn!

- Trick
	- reorder last two examples
	- what does this show?