# Machine learning: match the validation unit to the prediction task — Study worksheet [Study guides](https://unireadinglist.com/guides/machine-learning/) [Course topic: Machine Learning and Artificial Intelligence](https://unireadinglist.com/subjects/computer-science/#course-machine-learning-ai) Sources checked: 2026-11-04 Study route and exercise: University Reading List editorial project. For learners with basic statistics, regression, Python reading and distance calculations. ISLP offers the accessible applied spine; Murphy needs stronger linear algebra, probability and calculus; AIMA broadens the question to agents and decision problems. Every device, feature and label below is synthetic, with no population-performance or causal claim. ## Choose your starting point - [ ] I want free applied statistical learning Use the Python ISL volume for concepts paired with labs. Begin with prediction error and resampling before comparing increasingly flexible methods. - [ ] I need a probabilistic foundation Use Murphy after the mathematical prerequisites. Ask what joint distribution and decision rule a model assumes, not merely which package implements it. - [ ] I need the wider AI map Use AIMA to distinguish learning a predictor from search, planning and agent decisions. Select learning chapters for the worksheet rather than treating every AI topic as supervised classification. ## Compare the three books ### An Introduction to Statistical Learning: with Applications in Python Gareth James; Daniela Witten; Trevor Hastie; Robert Tibshirani; Jonathan Taylor · 2023 Free applied spine Choose it when: You can read basic Python and statistics and want conceptual explanations paired with executable labs. Use another route when: You need every derivation or a first Python course before beginning the statistical questions. Selected reading: 2023 Python volume: Statistical Learning, Classification and Resampling Methods, then selected model-selection discussions. Limits and trade-offs: It is distinct from the 2013/2021 R volumes. Authors provide a free PDF; the publisher’s print/e-book formats and notebook dependencies remain separate objects. #### Try before choosing Write two legitimate validation questions for the same repeated-device table and choose a split unit for each. Check your work: Withholding a row is not the same as withholding the device that generated it. Your notes: ________________________________ ISBN: 9783031387463 [Official free reading](https://www.statlearning.com/) [Book record and access options](https://unireadinglist.com/subjects/computer-science/an-introduction-to-statistical-learning-with-applications-in-python/) ### Probabilistic Machine Learning: An Introduction Kevin P. Murphy · 2022 Free mathematical companion Choose it when: You want to connect prediction to probability, optimization and explicit decision criteria. Use another route when: Linear algebra, multivariable calculus and conditional probability are still unfamiliar. Selected reading: 2022 introductory volume: mathematical background, supervised learning and model assessment by title, before specialized transfer or deep-learning topics. Limits and trade-offs: The selected hardcover ISBN 9780262046824 identifies the 2022 book; the author’s free draft was updated in 2025 under noncommercial/no-derivatives terms. It is not the Advanced Topics sequel. #### Try before choosing State whether a future query is conditional on a known device or drawn from a new device, then name what independence your evaluation needs. Check your work: Identical feature rows from one group do not supply independent evidence about an unseen group. Your notes: ________________________________ ISBN: 9780262046824 [Official free reading](https://probml.github.io/pml-book/book1.html) [Book record and access options](https://unireadinglist.com/subjects/computer-science/probabilistic-machine-learning-murphy/) ### Artificial Intelligence: A Modern Approach Stuart Russell; Peter Norvig · 4 · 2020 Broad AI reference Choose it when: You need to situate prediction within agents, search, reasoning and action. Use another route when: You want a short statistical-learning lab sequence without selecting chapters. Selected reading: Fourth US edition: Intelligent Agents and Learning from Examples, then probabilistic/deep/reinforcement learning as relevant. Limits and trade-offs: The US and Global contents differ. Public exercises and code complement the commercial book; they do not make the complete book or all instructor solutions free. #### Try before choosing Describe an agent that uses the classifier, distinguishing its percept, predicted label, action and evaluation criterion. Check your work: A classification accuracy is not automatically an agent’s utility or a safety/decision guarantee. Your notes: ________________________________ ISBN: 9780134610993 [Book record and access options](https://unireadinglist.com/subjects/computer-science/artificial-intelligence-a-modern-approach/) ## Check your preparation ### What is fitted? Your notes: ________________________________ ### What is the target unit? Your notes: ________________________________ ### What does a score establish? Your notes: ________________________________ ## A six-step study plan - [ ] Define the target Selected reading: ISLP statistical-learning introduction. What to do: Write what a future observation represents and what information training already contains about it. Check your work: Make known-device and new-device targets separate sentences. Your notes: ________________________________ - [ ] Specify the rule Selected reading: Classification reading. What to do: State feature, distance,1-neighbor vote and deterministic tie handling. Check your work: Device ID is used to form folds, not as an extra prediction feature. Your notes: ________________________________ - [ ] Audit row withholding Selected reading: Resampling discussion and the original table. What to do: List the training neighbor used for each withheld row. Check your work: Record whether a same-device twin remains in training. Your notes: ________________________________ - [ ] Audit group withholding Selected reading: The same model-assessment reading. What to do: Remove both rows of a device and calculate its prediction. Check your work: Keep the algorithm fixed so the comparison isolates the withholding unit. Your notes: ________________________________ - [ ] Change deployment assumptions Selected reading: The solved follow-up. What to do: Prove a nearest-neighbor margin for bounded new readings of known devices. Check your work: A conditional robustness proof does not rescue an estimate for new devices. Your notes: ________________________________ - [ ] Report without leaking a claim Selected reading: Murphy model assessment or AIMA evaluation discussion. What to do: Report split unit, target, predictions and exact finite scores together. Check your work: Avoid selecting the best-looking score after seeing answers or promoting it to an unsupported generalization guarantee. Your notes: ________________________________ ## Worked exercise: Original repeated-device table: two different validation questions Four synthetic devices A, B, C, D have feature values 0,10,20,30 and stable labels 0,1,0,1. Each appears in two identical rows. Use 1-nearest-neighbor on the scalar feature, absolute distance, and choose the lower feature value on a tie. Compare leaving out one row at a time with leaving out both rows of a device. Assumptions: The eight rows are complete and exact. Device ID defines grouping but is not a prediction feature. Train only on the retained rows, with the same fixed rule in every fold. Scores are counts of correct withheld-row predictions, not estimates with supplied sampling guarantees. Your notes: ________________________________ ## Change an assumption: Changed target: bounded readings of known devices Now deploy only on the same four known devices, retain all eight training rows, and stipulate each query’s label stays its device label. Its feature differs from that device’s original value by at most ε. First set ε=1, then find a sufficient uniform radius and test ε=5. Your notes: ________________________________ ## Before moving on - I can define the target unit and construct the matching folds. - I can reproduce both prediction ledgers and the strict distance bound. - I can distinguish a finite score, a conditional robustness proof and a population claim. ## Answer and interpretation ### What is fitted? The worksheet fixes 1-nearest-neighbor and its tie rule before evaluation; there is no hyperparameter selection or preprocessing to tune on withheld rows. ### What is the target unit? It may be another reading of an already represented device or a device absent from training. The correct withholding unit depends on that target. ### What does a score establish? An exact result on the supplied finite packet. Estimating future-population performance requires a justified sampling/deployment relation and uncertainty analysis beyond the table. ### Original repeated-device table: two different validation questions 1. Row-wise withholding leaves the identical same-device twin at distance 0. Every withheld row receives its correct label, giving 8/8=100%. 2. Withhold device A: its nearest retained feature 10 has label 1, wrong for A 0. Withhold D: nearest 20 has label 0, wrong for D 1. 3. Withhold B: features 0 and 20 tie, both label 0, so its prediction is wrong. Withhold C:10 and 30 tie, both label 1, also wrong. Both rows of every held-out device fail:0/8=0%. 4. Neither arithmetic score is broken. They withhold different information. The row score measures this packet’s already-represented-device task, while the group score removes device-specific representation. Neither alone proves a population accuracy. Row withholding 100%; whole-device withholding 0% on these eight invented rows. A split unit must match the intended prediction question. Test the result: List all eight predictions under both schemes and the neighbor used. Do not infer that grouped validation always produces zero or that row validation is always inappropriate. ### Change an assumption: Changed target: bounded readings of known devices 1. At ε=1 the same-device training feature is at distance at most 1. Any other device’s feature is at least 10−1=9 away, so every allowed query is correctly classified. 2. For general 0≤ε\<5, the own-device distance is at most ε while every other-device distance is at least 10−ε\>ε. This proves a uniform guarantee under the stated target and stable-label assumptions. 3. At ε=5 the guarantee fails: device B can produce query 5 with true label 1. Features 0 and 10 tie; the lower-feature tie rule selects 0, predicting label 0. One valid counterexample is enough to refute a guarantee for all devices. All allowed known-device queries are correct at ε=1, and uniformly for ε\<5. Equality 5 does not give the same all-device guarantee. This is a conditional distance proof, not a claim about unknown devices. Test the result: Check the strict inequality and the B-at 5 counterexample. If labels drift, devices are new, or the distance rule changes, the supplied proof no longer establishes the result. ## Common mistakes ### Row independence assumed Repeated rows can retain exactly the device-specific information the deployment task would withhold. ### Prediction confused with action quality A score for labels does not specify losses, decisions or outcomes for an agent. ### Finite packet promoted to a guarantee Exact calculations establish these rows or the explicit distance assumptions, not an unspecified future distribution. ## Sources and edition checks - [Authors: free ISL Python volume and edition distinctions](https://www.statlearning.com/) The authors distinguish Python 2023 from the R editions and provide a free applied reading route. - [Springer:2023 Python book and five authors](https://link.springer.com/book/10.1007/978-3-031-38747-0) The publisher identifies the five-author Python volume and its prediction/resampling scope. - [Author: introductory volume and updated free draft](https://probml.github.io/pml-book/book1.html) The free updated introductory draft and its permissions are distinct from the printed publication date. - [MIT Press:2022 introductory hardcover](https://mitpress.mit.edu/9780262046824/probabilistic-machine-learning/) The 2022 hardcover and Advanced Topics sequel are different works. - [Authors: fourth US edition, agents and learning contents](https://aima.cs.berkeley.edu/) The fourth US edition supplies a broad agent-and-learning comparison, not a substitute for the same statistical-lab sequence. These are proposed study activities, not official course assignments or a claim that every book has been read cover to cover. [How we curate](https://unireadinglist.com/editorial/)