This blog is a companion to my recent book, Exploring Data in Engineering, the Sciences, and Medicine, published by Oxford University Press. The blog expands on topics discussed in the book, and the content is heavily example-based, making extensive use of the open-source statistical software package R.

## Sunday, March 9, 2014

### A question of model uncertainty

It has been several months since my last post on classification tree models, because two things have been consuming all of my spare time.  The first is that I taught a night class for the University of Connecticut’s Graduate School of Business, introducing R to students with little or no prior exposure to either R or programming.  My hope is that the students learned something useful – I can say with certainty that I did – but preparing for the class and teaching it took a lot of time.  The other activity, that has taken essentially all of my time since the class ended, is the completion of a book on nonlinear digital filtering using Python, joint work with my colleague Moncef Gabbouj of the Tampere University of Technology in Tampere, Finland.  I will have more to say about both of these activities in the future, but for now I wanted to respond to a question raised about my last post.

Specifically, Professor Frank Harrell, the developer of the extremely useful Hmisc package, asked the following:

How did you take into account model uncertainty?  The uncertainty resulting from data mining to find nodes and thresholds for continuous predictors has a massive impact on confidence intervals for estimates from recursive partitioning.

The short answer is that model uncertainty was not accounted for in the results I presented last time, primarily because – as Professor Harrell’s comments indicate – this is a complicated issue for tree-based models.  The primary objective of this post and the next few is to discuss this issue.

So first, what exactly is model uncertainty?  Any time we fit an empirical model to data, the results we obtain inherit some of the uncertainty present in the data.  For the specific example of linear regression models, the magnitude of this uncertainty is partially characterized by the standard errors included in the results returned by R’s summary() function.  This magnitude depends on both the uncertainty inherent in the data and the algorithm we use to fit the model.  Sometimes – and classification tree models are a case in point – this uncertainty is not restricted to variations in the values of a fixed set of parameters, but it can manifest itself in substantial structural variations.  That is, if we fit classification tree models to two similar but not identical datasets, the results may differ in the number of terminal nodes, the depths of these terminal nodes, the variables that determine the path to each one, and the values of these variables that determine the split at each intermediate node.  This is the issue Professor Harrell raised in his comments, and the primary point of this post is to present some simple examples to illustrate its nature and severity.

In addition, this post has two other objectives.  The first is to make amends for a very bad practice demonstrated in my last two posts.  Specifically, the classification tree models described there were fit to a relatively large dataset and then evaluated with respect to that same dataset.  This is bad practice because it can lead to overfitting, a problem that I will discuss in detail in my next post.  (For a simple example that illustrates this problem, see the discussion in Section 1.5.3 of Exploring Data in Engineering, the Sciences, and Medicine.)  In the machine learning community, this issue is typically addressed by splitting the original dataset randomly into three parts: a training subset (Tr) used for model-fitting, a validation subset (V) used for intermediate modeling decisions (e.g., which variables to include in the model), and a test subset (Te) used for final model evaluation.  This approach is described in Section 7.2 of The Elements of Statistical Learning by Hastie, Tibshirani, and Friedman, who suggest 50% training, 25% validation, and 25% test as a typical choice.

The other point of this post is to say something about the different roles of model uncertainty and data uncertainty in the practice of predictive modeling.  I will say a little more at the end, but whether we are considering business applications like predicting customer behavior or industrial process control applications to predict the influence of changes in control valve settings, the basic predictive modeling process consists of three steps: build a prediction model; fix (i.e., “finalize”) this model; and apply it to generate predictions from data not seen in the model-building process.  In these applications, model uncertainty plays an important role in the model development process, but once we have fixed the model, we have eliminated this uncertainty by fiat.  Uncertainty remains an important issue in these applications, but the source of this uncertainty is in the data from which the model generates its predictions and not in the model itself once we have fixed it.  Conversely, as George Box famously said, “all models are wrong, but some are useful,” and this point is crucial here: if the model uncertainty is great enough, it may be difficult or impossible to select a fixed model that is good enough to be useful in practice.

Returning to the topic of uncertainty in tree-based models, the above plot is a graphical representation of a classification tree model repeated from my previous two posts.  This model was fit using the ctree procedure in the R package party, taking all optional parameters at their default values.  As before, the dataset used to generate this model was the Australian vehicle insurance dataset car.csv, obtained from the website associated with the book Generalized Linear Models for Insurance Data, by Piet de Jong and Gillian Z. Heller.  This model – and all of the others considered in this post – was fit using the same formula as before:

Fmla = clm ~ veh_value + veh_body + veh_age + gender + area + agecat

Each record in this dataset describes a single-vehicle, single-driver insurance policy, and clm is a binary response variable taking the value 1 if policy filed one or more claims during the observation period and 0 otherwise.  The other variables (on the right side of “~”) represent covariates that are either numeric (veh_value, the value of the vehicle) or categorical (all other variables, representing the vehicle body type, its age, the gender of the driver, the region where the vehicle is driven, and the driver’s age).

As I noted above, this model was fit to the entire dataset, a practice that is to be discouraged since it does not leave independent datasets of similar character for validation and testing.  To address this problem, I randomly partitioned the original dataset into a 50% training subset, a 25% validation subset, and a 25% test subset as suggested by Hastie, Tibshirani and Friedman.  The plot shown below represents the ctree model we obtain using exactly the same fitting procedure as before, but applied to the 50% random training subset instead of the complete dataset.  Comparing these plots reveals substantial differences in the overall structure of the trees we obtain, strictly as a function of the data used to fit the models.  In particular, while the original model has seven terminal nodes (i.e., the tree assigns every record to one of seven “leaves”), the model obtained from the training data subset has only four.  Also, note that the branches in the original tree model are determined by the three variables agecat, veh_body, and veh_value, while the branches in the model built from the training subset are determined by the two variables agecat and veh_value only.

These differences illustrate the point noted above about the strong dependence of classification tree model structure on the data used in model-building.  One could object that since the two datasets used here differ by a factor of two in size, the comparison isn’t exactly “apples-to-apples.”  To see that this is not really the issue, consider the following two cases, based on the idea of bootstrap resampling.  I won’t attempt a detailed discussion of the bootstrap approach here, but the basic idea is to assess the effects of data variability on a computational procedure by applying that procedure to multiple datasets, each obtained by sampling with replacement from a single source dataset.  (For a comprehensive discussion of the bootstrap and some of its many applications, refer to the book Bootstrap Methods and their Application by A.C. Davison and D.V. Hinkley.)  The essential motivation is that these datasets – called bootstrap resamples – all have the same essential statistical character as the original dataset.  Thus, by comparing the results obtained from different bootstrap resamples, we can assess the variability in results for which exact statistical characterizations are either unknown or impractical to compute.  Here, I use this idea to obtain datasets that should address the “apples-to-apples” concern raised above.  More specifically, I start with the training data subset used to generate the model described in the previous figure, and I use R’s built-in sample() function to sample the rows of this dataframe with replacement.  For an arbitrary dataframe DF, the code to do this is simple:

> set.seed(iseed)
> BootstrapIndex = sample(seq(1,nrow(DF),1),size=nrow(DF),replace=TRUE
> ResampleFrame = DF[BootstrapIndex,]

The only variable in this procedure is the seed for the random sampling function sample(), which I have denoted as iseed.  The extremely complicated figure below shows the ctree model obtained using the bootstrap resample generated from the training subset with iseed = 5.

Comparing this model with the previous one – both built from datasets of the same size, with the same general data characteristics – we see that the differences are even more dramatic than those between the original model (built from the complete dataset) and the second one (built from the training subset).  Specifically, while the training subset model has four terminal nodes, determined by two variables, the bootstrap subsample model uses all six of the variables included in the model formula, yielding a tree with 16 terminal nodes.  But wait – sampling with replacement generates a significant number of duplicated records (for large datasets, each bootstrap resample contains approximately 63.2% of the original data values, meaning that the other 36.8% of the resample values must be duplicates).  Could this be the reason the results are so different?  The following example shows that this is not the issue.

This plot shows the ctree model obtained from another bootstrap resample of the training data subset, obtained by specifying iseed = 6 instead of iseed = 5.  This second bootstrap resample tree is much simpler, with only 7 terminal nodes instead of 16, and the branches of the tree are based on only four of the prediction variables instead of all six (specifically, neither gender nor veh_body appear in this model).  While I don’t include all of the corresponding plots, I have also constructed and compared the ctree models obtained from the bootstrap resamples generated for all iseed values between 1 and 8, giving final models involving between four and six variables, with between 7 and 16 terminal nodes.  In all cases, the datasets used in building these models were exactly the same size and had the same statistical character.  The key point is that, as Professor Harrell noted in his comments, the structural variability of these classification tree models across similar datasets is substantial.  In fact, this variability of individual tree-based models was one of the key motivations for developing the random forest method, which achieves substantially reduced model uncertainty by averaging over many randomly generated trees.  Unfortunately, the price we pay for this improved model stability is a complete loss of interpretibility.  That is, looking at any one of the plots shown here, we can construct a simple description (e.g., node 12 in the above figure represents older drivers – agecat > 4 – with less expensive cars, and it has the lowest risk of any of the groups identified there).  While we may obtain less variable predictions by averaging over a large number of these trees, such simple intuitive explanations of the resulting model are no longer possible.

I noted earlier that predictive modeling applications typically involve a three-step strategy: fit the model, fix the model, and apply the model.  I also argued that once we fix the model, we have eliminated model uncertainty when we apply it to new data.  Unfortunately, if the inherent model uncertainty is large, as in the examples presented here, this greatly complicates the “fix the model” step.  That is, if small variations in our training data subset can cause large changes in the structure of our prediction model, it is likely that very different models will exhibit similar performance when applied to our validation data subset.  How, then, do we choose?  I will examine this issue further in my next post when I discuss overfitting and the training/validation/test split in more detail.

1. I am trying to use CART algorithms as against conventional statistics, DOE and hypothesis testing. I have got some results and advantages also for problem solving rather than using the data as training set for future prediction.

Is my understanding correct that uncertainty is more important in cases where we use this training data rather than problem solving approach?

2. It looks like you spend a lot of effort and time on your blog. I have bookmarked it and I am looking forward to reading new articles. Keep up the good work
Android Training in Chennai

3. I am expecting more interesting topics from you. And this was nice content and definitely it will be useful for many people.

4. Wow it is really wonderful and awesome thus it is very much useful for me to understand many concepts and helped me a lot.thus these tips are really awesome and you had a wonderful products.

Online Reputation Management

5. great tips thanks for your impressive enhancement of getting unique style of design a website. and it much more wonderful information to me. keep share many different ideas.
IELTS coaching in chennai

6. Truly a very good article on how to handle the future technology. After reading your post,thanks for taking the time to discuss this, I feel happy about and I love learning more about this topic
Tooth Braces In Chennai

7. wonderful post and very helpful, thanks for all this information. You are including better information regarding this topic in an effective way.Thank you so much
Web Design Company in Chennai

8. Thank you for sharing such a nice and interesting blog with us. But in your blog, I had a chance to get some useful and unique information. I would like to suggest your blog.
Web development Company in Chennai

9. this is very nice blog this studying course information very useful to everyone who have learning this information.

Base SAS Training in chennai

10. This is a nice article here with some useful tips for those who are not used-to comment that frequently.Thanks for this helpful information I agree with all points you have given to us.I will follow all of them.

Self Employment Tax
Tax Preparation Services
Tax Accountant
Tax Consultant
Online Tax services
Tax Professional
Online Tax Preparation"

11. It's Really nice post.Thank you for sharing such a blog....!!
Spoken english Training in Bangalore

12. usefull and awesome article.

14. very nice blog thanks for sharing

15. very nice blog It was really useful.
Best Oracle Training in Bangalore

16. These ways are very simple and very much useful, as a beginner level these helped me a lot thanks fore sharing these kinds of useful and knowledgeable information.

17. Thanks for giving this information.
SAP Basis Training in Chennai

18. useful and knowledgeable information.
thank u sharing!!

This blog was very useful for me waiting for more blog.

SAP FICO Training in Chennai

19. It is imperative and useful......Keep it up

Best Training Institute in Chennai

20. It is really very excellent. Because of all given information was wonderful and it's very helpful for me. Thanks for sharing
SAP Training in Chennai
SAP ABAP Training in Chennai
SAP FICO Training in Chennai
SAP MM Training in Chennai

21. These ways are very simple and very much useful, as a beginner level these helped me a lot thanks fore sharing these kinds of useful and knowledgeable information.
Also Check out the : https://www.credosystemz.com/training-in-chennai/best-data-science-training-in-chennai/

22. Benefits of reading your post. This will help in the maintenance of our health.The daily routine will help you lose weight quickly and safely.My life is totally transformed after I followed this diet.I feeling great about myself.

Herbalife in Chennai
Herbalife Nutrition Products
Nutritionclub in Chennai
Weight Loss in Chennai
Weight Gain in Chennai

23. Your site is amazing and your blogs are informative and knlowledgeble to my websites.This is one of the best tips in my life.I have in quite some time.Nicely written and great info.Thanks to share the more informations.

Seo Experts
Seo Company
Web Designing Company
Digital Marketing
Web Development Company
Apps Development

24. This is a great post. I like this topic.This site has lots of advantage. I found many interesting things from this site. It helps me in many ways.Thanks for posting this again.
Best ERP Software
ERP Software In India
ERP Companies In Chennai
ERP Software In Chennai

25. Amazing article. Your blog helped me to improve myself in many ways thanks for sharing this kind of wonderful informative blogs in live. I have bookmarked more article from this website. Such a nice blog you are providing ! Kindly Visit Us

R Programming institutes in Chennai | R Programming Training in Chennai | R Programming Course Fees | R Language training in Chennai

26. Great post! I am actually getting ready to across this information, It's very helpful for this blog.Also great with all of the valuable information you have Keep up the good work you are doing well. big data training in Velachery | Hadoop Training in Chennai | big data Hadoop training and certification in Chennai | Big data course fees |

27. Really superb!!! I read your blog regularly and your content is truly good. I thank you for your effective and useful post.
IELTS coaching in Chennai
IELTS coaching centre in Chennai
IELTS Training in Chennai
Best IELTS coaching in Chennai
Best IELTS coaching centres in Chennai
IELTS classes in Chennai

28. I feel happy about and learning more about this topic. keep sharing your information regularly for my future reference. This content creates new hope and inspiration within me. Thanks for sharing an article like this. the information which you have provided is better than another blog.
IELTS Coaching in Dwarka mor

29. Great Post,really it was very helpful for us.
Thanks a lot for sharing!
I found this blog to be very useful!!

30. Vanskeligheter( van bi ) vil passere. På samme måte som( van điện từ ) regnet utenfor( van giảm áp ) vinduet, hvor nostalgisk( van bướm ) er det som til slutt( van cửa ) vil fjerne( van công nghiệp ) himmelen.

31. Si el agua cae al lago, desaparecerá( phụ kiện tủ bếp ). Pero si cae a la hoja de( phụ kiện tủ áo ) loto, brillará como una joya. Caer igual pero( thùng gạo thông minh ) estar con alguien es importante.

32. Good blog article
Sanjary Kids is one of the best play school and preschool in Hyderabad,India. Give your child the best preschool experience by choosing the best playschool of Hyderabad in Abids. we provide programs like Play group,Nursery,Junior KG,Senior KG,and provides Teacher Training Program.

33. Good blog posts information by the author

Sanjary Academy is the best Piping Design institute in Hyderabad, Telangana. It is the best Piping design Course in India and we have offer professional Engineering Courses like Piping design Course, QA/QC Course, document controller course, Pressure Vessel Design Course, Welding Inspector Course, Quality Management Course and Safety Officer Course.
Piping Design Course
Piping Design Course in Hyderabad ­
Piping Design Course in India­

Sanjary Academy is the best Piping Design institute in Hyderabad, Telangana. It is the best Piping design Course in India and we have offer professional Engineering Courses like Piping design Course, QA/QC Course, document controller course, Pressure Vessel Design Course, Welding Inspector Course, Quality Management Course and Safety Officer Course.
Piping Design Course
Piping Design Course in Hyderabad ­
Piping Design Course in India­

Sanjary Academy provide pressure vessel design,quality management system course, piping design course, qa/qc course and document controller course.
Welding Inspector Course
Safety officer course
Quality Management Course
Quality Management Course in India

36. Thanks for sharing information
"Sanjary Academy provides excellent training for Piping design course. Best Piping Design Training Institute in Hyderabad,
Telangana. We have offer professional Engineering Course like Piping Design Course,QA / QC Course,document Controller
course,pressure Vessel Design Course, Welding Inspector Course, Quality Management Course, #Safety officer course."
Piping Design Course in India­
QA / QC Course
QA / QC Course in india
QA / QC Course in Hyderabad
Document Controller course
Pressure Vessel Design Course
Welding Inspector Course
Quality Management Course
Quality Management Course in india
Safety officer course

37. Nice blog..! I really loved reading through this article. Thanks for sharing.You done a great job.Oracle Applications Training in Bangalore

38. Thank you for excellent article.You made an article that is interesting.
Digital marketing course in Bangalore with Live Projects. 100% placement, 20+ modules, 10+ certifications, Great discounts on course fees.
https://onlineidealab.com/digital-marketing-courses-in-bangalore/

39. Thanks for one marvelous posting! I enjoyed reading it; you are a great author. I will make sure to bookmark your blog and may come back someday. I want to encourage that you continue your great posts.

learn digital marketing

40. Awesome article, it was exceptionally helpful! I simply began in this and I'm becoming more acquainted with it better! Cheers, keep doing awesome
All University BCOM 1st Year TimeTable 2020

41. This was definitely one of my favorite blogs. Every post published did impress me. ExcelR Digital Marketing Class In Pune

42. I concur with a ton of the focuses you made in this article. I value the work you have placed into this and expectation you keep composing regarding this matter.
Best Data Science training in Mumbai

Data Science training in Mumbai

43. Thanks really it is very useful blog obia online training Evanta technologies provide you the best OBIA online training with step by step procedure for installing, customizing and configuring of OBIA.

44. I am glad to read that you come up with outstanding information that definitely allows me to share with others. Thank you for sharing this with us.

digital marketing coaching in hubli

45. Your info is really amazing with impressive content..Excellent blog with informative concept. Really I feel happy to see this useful blog, Thanks for sharing such a nice blog..
Android Training Institute in Chennai | Android Training Institute in anna nagar | Android Training Institute in omr | Android Training Institute in porur | Android Training Institute in tambaram | Android Training Institute in velachery

46. It is a very helpful and informative blog post. I would like to thank to you for providing such information I have also have a website providing very good information.good job ..keep it up
Ai & Artificial Intelligence Course in Chennai
PHP Training in Chennai
Ethical Hacking Course in Chennai Blue Prism Training in Chennai
UiPath Training in Chennai

47. Thanks for writing this article such a good...it helps to all people for our health related problems like muscle pain, chronic pain, or joint pain and so on...
Order Soma 500mg cash on delivery in cheap price
Order Soma 500mg Online COD with PayPal and Credit/Debit Card

48. Know about Buying Tramadol 100mg Cash On Delivery. Get Tramadol In Best 2020 Price Online thru Cash On Delivery C.O.D option online.you can order Tramadol 100mg in USA cash on delivery facility.Order Tramadol Online thru Cash on delivery option. Doctors generally prescribe this medication for the treatment of moderate and moderately severe pain. Order Tramadol COD online now in cheapest price with fast and secure Cash on Delivery.

49. Really amazing experience to visit your blog, thanks a lot for sharing with us. Visit Ogen Infosystem for creative website designing and ppc services at best price in Delhi.
Web Designing Company

50. Hi
I visited your blog you have shared amazing information, i really like the information provided by you, You have done a great work. I hope you will share some more information regarding full movies online. I appreciate your work.
Thanks

OpenStack Training in Bangalore

51. Thanks mate. I am really impressed with your writing talents and also with the layout on your weblog. Appreciate, Is this a paid subject matter or did you customize it yourself? Either way keep up the nice quality writing, it is rare to peer a nice weblog like this one nowadays. Thank you event marketing and How to Write a Speaker Biography for a Conference

52. Very interesting article to read it. I would like to thank you for the efforts you had made for writing this wonderful article. This article inspired me to read more. Keep sharing on updated posts…

Learn Digital Marketing Course in Bangalore with Live Project Work & Case Studies taught by Ranjan Jena (10Yrs Trainer). 100% Guarantee to Clear Job Interview

Green Xanax Bars [S 90 3]
White Xanax Bars
Yellow Xanax Bars

54. Your blog provided us with valuable information to work with. Each & every tips of your post are awesome. Thanks a lot for sharing. Keep blogging,
Best Digital Marketing Institute in Hyderabad

55. So you can save your lot of time by hiring SEO Freelancer or if you want to learn full SEO Course Delhi

56. Thanks for the always useful information. This is great information to help peoples and nice article written by writer. CnX Player is a powerful & efficient 4K ultra HD enabled video player for Windows 10 PC & Tablet, Android and iOS – iPhone & iPad.

57. Thanks for posting the best information and the blog is very helpfuldigital marketing institute in hyderabad

58. This site helps to clear your all query.
This is really worth reading. nice informative article.
uttar matric scholarship

59. nice informative post. Thanks you for sharing.
Wordpress Development
Mobile App Development

60. Thanks for choosing this specific Topic. As i am also one of big lover of this. Your explanation in this context is amazing. Keep posted these overwarming facts in front of the world.
Printer customer support number

61. Thanks for sharing.

ERP HR Module in Bangalore
Zenetial is the best HR module in ERP Bangalore. This ERP payroll software manages all your employee details from hire to retire. Get a free demo of HR software.

62. Thanks for sharing.
Are you are looking for aeronautical engineering for studying in Bangalore. We can proudly say we are the best aeronautical admissions providers in top colleges and universities.

63. IELTS is known for the best IELTS coaching in Ahmedabad and consistently successful results. High Band Score 8 and Average Band Score Minimum 6.8 are maintained so that you can get more information about IELTS Coaching Institute, Ahmedabad.

ILETS is known for the best quality ILETS coaching and consistently successful results. Maintaining the highest band score of 8 and the average band scores minimum of 6.8 to help you to learn more about the IELTS Institute of Ahmedabad

64. Amazing write-up! Really Good.
Now the foundation of any business is marketing. Adsify marketing is the best Digital marketing in Trivandrum.

65. they are very simple and usefull information thanksfor sharing

66. Thanks for the blog but If your looking to switch your career to the Airline Industry we the Fusion is the best place to pursue. It is one of the best Cabin Crew Training Centre

67. Thank you for taking the time to provide us with your valuable information.
Cheapest email marketing Services In India

68. It is the intent to provide valuable information and best practices, including an understanding of the regulatory process. business analytics course in kanpur

69. Excellent information providing by your post, thank you
Best HIV Treatment Hospital in India | Latest HIV Treatment in India

70. Great! Thanks for sharing this kind of information it’s really very nice and useful for me.
Website Designing Company in South Delhi
Web Plus Era is a leading Website Designing Company in Dwarka. We offer web development solutions, but also Delhi web services. Mobile apps have been the center of attraction for most of the people. For almost all e-commerce websites and business operators, mobile apps are a wonderful way to make your way to a lot of major customers, clients, and a large market segment. Web Plus Era is the best App Development Company in Delhi. Responsive Website designs are in excessive call for because of their better functionalities and responsiveness to make stop person revel in a memorable one. Companies are making an investment large on responsive net layout offerings. Web Plus Era is among the quickest rising Responsive Web Design Services responsive net designing organizations within side the India,Delhi. Website Designing Company in DelhiVery satisfied with the Website designing work effort by your team! Happy to recommend it to anyone looking for Website Designing service Web Plus Era is best Place for your Digital Work like website,seo,smo etc.

71. Thanks for sharing such an informational blog which will surely be a big help to the small medium enterprise so that they can choose the best suited tool for their business.
top 10 crm software in india

72. Thanks for Nice and Informative Post. Great info!
clean power for the planet

73. ï»¿great post, I enjoy it. I also buy ambien a lot. You need to buy ambien as well. I will travel to the whole USA and buy ambien there and in other areas to. i will be downtown, you know for what ? just to buy ambien. I will sleep a ton immediately after buy ambien and that it usually make me feel so very good. I will be buy ambien in a pharmacy or a different locations, no matter what the existence consider me . Judi Online

74. Dead pent subject material , Really enjoyed examining . hotmail

75. Thank you for giving the great article. It delivered me to understand several things about this concept. Keep posting such surpassing articles so that I gain from your great post.
ibm full form in india |
ssb ka full form |
what is the full form of dp |
full form of brics |
gnm nursing full form |
full form of bce |
full form of php |
bhim full form |
nota full form in india |
apec full form

76. good info about model uncertainity

77. good info about model uncertainity

78. Appreciation for sharing this bend towards. I in truth other than your weblog uncover no think about packs. you have in truth shared an illuminating and drawing in weblog dissipate following individuals.. Free Quick Heal Product Key

79. i'm glad to find this say incredibly profitable for me, since it involves part of are searching for. I consistently choose to affirmation the man or woman content and this case I found in you announce. much gratitude to you for sharing. Wifi Password Hacker For Pc

80. Ive been attempting to find a few respectable stuff concerning the challenge and dock't had any good fortune going on till this narrowing, You simply got a subsidiary biggest lover!.. Best Farewell Quotes For Friends

81. Awesome article! You are providing us very useful information. Thank you for sharing. Check out Digital Marketing Institute in Pune

82. I am really grateful for giving me an opportunity to read a good content.
Digital marketing courses in New zealand

83. This blog is a simple plus has a detailed analysis on the Search Engine Marketing. Which gives us the advantage plus the disadvantages of various SEM's. To know more do visit -
Search Engine Marketing

84. Thank you for sharing your knowledge in this blog. It will help many people in exploring data area. Keep posting. Content Writing Course in Bangalore

85. Your article has to be one of the most detailed blogs out there for the kind of information provided, keep publishing more articles to spread more knowledge. And those who are searching for Digital Marketing Courses in Nigeria can refer the following blog, thank you.

Digital marketing courses in Nigeria

86. Nice and detailed post. Looking to learn digital marketing in Dehradun with hands on training by the industry experts then visit us: Digital Marketing Course in Dehradun

87. I was looking for this exact info and thankfully I came across this article. Digital marketing courses in Ahmedabad

88. A very descriptive content with infographics and illustrated manner. Explained very well on the topic. Thanks for sharing. If someone is looking for Digital Marketing Course in France then follow the link and go through to get the entire details of the course and other courses as well. This is the full power-packed course content you can acquire great expertise by joining our comprehensive course.
Digital marketing courses in france

89. Thank you for the information
I concur that this is the most thorough description of the subject. I'm so glad I found your blog and am looking forward to reading your future posts. And I have referred to related content in the link below.

. We provide a variety of software-related courses along with complete placement assistance.

Excellent IT instruction has been given to our pupils in a number of methods by our teachers and subject-matter specialists.

We offer top-notch instruction in Android technologies, and we frequently update our curricula to include the most recent IT trends.

We provide placement help for recent grads (recent graduates). We will offer support to all eligible applicants.

Contact Details:
Address: No-06, Ground Floor, 5th Main Road, Vijaya Nagar Velachery, Chennai – 600042.
Phone: 6385872810

90. The article is well informed with logical ideas and also it is helpful as well. Digital marketing courses in Agra

92. Great informative blog. This is a valuable source of information. Thanks for sharing your knowledge over here. Please check the Digital Marketing Courses in Delhi to know more about the topic. Surely you already know the power of Digital Marketing when you want to boost your website, your career or your business. Read more here: Digital Marketing Courses in Delhi

93. This is by far one of the most engaging articles I have read in recent times. Just loved the quality of information provided and I must say you have noted down the points very precisely, keep posting more. Digital Marketing is now booming at a rapid pace, especially in Dubai, and many are now searching for the courses. So to ease their work I am leaving a link below for those who are searching for Digital Marketing courses in Abu Dhabi. All the best and keep learning, thank you.
Digital Marketing Courses in Abu Dhabi

94. Excellent blog. The time and efforts you have put to make this can be clearly seen. Thank you for sharing this with us.

Looking to learn about digital marketing in nagpur then check out this informative article Digital marketing courses in Nagpur it provides all the valuable information you need.

95. Highly appreciable content!
If anyone is interested in building a medical career but are struggling to clear medical entrance exams, Wisdom Academy is the right place to begin. It is one of Mumbai’s best NEET coaching institutes for students preparing for medical and other competitive-level entrance examinations. It offers comprehensive learning resources, advanced study apparatus, doubt-clearing sessions, mentoring, and much more. Enroll Now!
NEET Coaching in Mumbai

Digital marketing courses in Noida

97. I read your article and I really appreciate the fact you have given so much effort in sharing with us this good knowledge. Keep growing and keep sharing with us such lovely contents.
Also do visit to our Digital Marketing courses in Bahamas

98. Good Works! Its a great pleasure reading your post.Its full of information I am looking for and I love to post a comment that “The content of your post is awesome” Great work. Content Writing Courses in Delhi

99. Worth reading article. This is a nice post for those who are interested in Data in Engineering. Thanks for sharing it here. We also provide an informational and educational blog about Freelancing. Nowadays, many people want to start a Freelance Career without knowing How and Where to start. People are asking:
What is Freelancing and How Does it work?
How to Become a Freelancer?
Is working as a Freelancer a good Career?
How much can a Freelancer earn?
Can I live with a Self-Employed Home Loan?
What Kind of Freelancing Jobs can I find?
Which Freelancers Skills are required?
How to get Freelance projects?
How Do companies hire Freelancers?
In our Blog, you will find a guide with Tips and Steps which will help you to take a good decision. Start reading and find out the Answers:
What is Freelancing

100. Nice article . Thank you for this beautiful content, Keep it up. This blog has clear about related to the model uncertainty with detailed information. Thanks again for sharing. Financial Modeling Courses in India

101. Very well explained! Keep posting more contents.
If you are searching for the ideal IAS coaching institute in Hyderabad, there is no better choice than Pragnya IAS Academy. It is one of the best coaching institutes in Hyderabad, offering civil services coaching.

102. model uncertainty is described with smart content and explanation is extremely wonderful. Its really my luck to have come across this blog. Its really a great content. Digital marketing courses in Kota

103. Great effort makes me understand the concept better.Thanks for sharing.
Are you looking for the best financial modeling courses in India? Financial modeling skills are increasingly important to upgrade your career and work in the ever-growing finance sector. This article lists the best colleges in India that provide financial modeling courses. Continue reading to choose the best course for you.
Financial Modeling Courses in India

104. Nice blog on model uncertainty
Digital Marketing Courses in Pune

105. About the article, I just wanted to say it is really nice and informative. Digital Marketing Courses in Faridabad

106. Truly great informative article on R. This most sought language and package is used most widely now for data mining and big analysis and data engineering. The article is very well described with diagrams and notes. Great work. Thanks for sharing. Keep sharing. I truly enjoyed the content by knowing deep insight and issues. If anyone wants to build his carrier in Digital Marketing then you must go through our curriculum which is designed very professionally with cutting edge of the current requirement of the corporates and based on market trends. You will be taught in a highly professional environment with practical assignments. You can decide your specialized stream your own way by understanding each subject in depth under the guidance of highly professional and experienced trainers. For more detail Please visit at
Digital Marketing Courses in Austria

107. What an splendid and articulate blog. Enjoyed reading it. Do share more articles. Thanks.

Digital marketing courses in Chennai

108. Truly a very good service with add-on features to make available the content to broader audience. Thanks for sharing the update on A question of model uncertainty. If anyone wants to learn Digital Marketing, please join the world-class curriculum and industry-standard professional skills. For more details, please visit Digital marketing Courses in Bhutan

109. Awesome article on uncertainty function and predictive modeling, very well explained.
Digital marketing courses in Raipur

110. I really appreciate your efforts in creating this incredible article, thank you for taking the time to explain the topic in detail. It's authors like you who encourage readers to keep reading and to learn more.
Data Analytics Courses In Kolkata

111. Your articles touch differently. I love your writing so much. keep share more article in future also. thank you so much.If someone is looking for data analytics courses in Indore then here is top 5 courses explained in this blog. Please check once for more information. Data Analytics Courses In Indore

112. very good article on uncertainty model. Thanks for sharing

113. This blog post discusses the different sources of model uncertainty and how they can be addressed. It is a well-written and informative post that will be helpful for those looking to learn more about this topic. Data Analytics Courses In Coimbatore

114. Hi, I have gone through few of your blogs and must say they are really interesting as well as helpful. This blog clearly shows the efforts put in by the blogger to come up with the excellent in detail algorithms. Looking forward to more of such blogs.
Data Analytics Courses In Kochi

115. Great topic about A question of model uncertainty. This post does a great job of outlining the different types of model uncertainty, and how they can impact the results of data analysis. Thanks for sharing with us Digital Marketing Courses in Australia

116. The article is worth reading every bit of it and gives a learning experience from this. Also, keep us updated about the post. Data Analytics Courses in Delhi

117. It's an excellent post. Nice blog on DataBlog. Fascinating to read this article. The way you have explained it is simple and understandable. The clear-cut description of General Linear Model and Bootstrap Methods. Thank you for the concise explanation and beneficial information. Well, I have gained a lot of understanding. Keep sharing more.
Digital marketing courses in Nagpur

118. Mind boggling blog post, the writer has done a great job in explaining the model uncertainty. Kudos to the writer! Data Analytics Courses In Coimbatore , This course is designed for students who want to pursue a career in data analytics or for those who want to gain a better understanding of how data analytics can be used to improve business decision making.

119. This is a very unique blog on the topic of A question of model uncertainty. I really liked reading and is one of the greatest read for me. Thanks for sharing this blog with us. Keep up the good work. Digital Marketing Courses in Vancouver

120. The various sources of model uncertainty are covered in this blog post, along with solutions. Those who want to learn more about this subject will find this to be a well-written and instructive post.
Data Analytics Courses in Ghana

121. This post is excellent. I like this information. The benefits and negatives of this website are numerous. This blog post contained a lot of valuable information. It benefits me greatly. I appreciate you reposting this.
Data Analytics Courses in Mumbai

122. This exploring data blog is just amazing that how data used in various fields. I am looking for more data blogs. Keep posting.  Data Analytics Courses In Vadodara

123. There's a lot to learn from this blog. Keep posting more blogs about data Digital marketing courses in Varanasi

124. Thanks for sharing the importance of data in the field of science, engineering, medicine. I would like to read more of this blog  Data Analytics Courses in navi Mumbai

125. Your blogs are simple & easy to understand. I really enjoyed reading this.  Data Analytics Courses In Bangalore

126. What a beautiful piece of writing! This is among the greatest articles I've seen regarding the model of uncertainty so far. Congratulations to the blogger for such a fantastic job! We are very grateful that you shared this important information with us.
Data Analytics Courses in Gurgaon

127. Nicely written, good blog. Keep sharing valuable content for references.
Data Analytics Courses in Visakhapatnam

128. Amazing blog. Learning programs such as R and Python is a challenging thing. But the way you have explained the base models is impressive. The in-depth discussion on variables and subsets helped to gain more understanding of the topic. Thanks for the article, and keep sharing more tech-oriented posts in the future.
Courses after bcom

129. Awesome blog. I enjoyed reading your article. I would like to thank you for sharing your content and ideas with yours. Keep up the good work.If your are looking for professional courses after senior secondary here is the list of top 20 professional courses please check on this link - Professional Courses

130. Great article very informative on Data Tree model of R language which is very effective and informative and described in a very descriptive manner with scripts and narratives. If anyone wants to build his carrier in Digital Marketing then you must go through our curriculum which is designed very professionally with cutting edge of the current requirement of the corporates and based on market trends. For more detail Please visit at
Digital marketing Courses In UAE

131. This fantastic blog explores the various sources of model uncertainty and how to deal with them. Those who want to learn more about this subject will find this to be a well-written and instructive post.
financial modelling course in kenya

132. I like the fact what you have acquired here like what you are stating and the way in which you have said it. You make it enjoyable and I have bookmarked it as well. Also, I am anticipating in perusing similar new articles. Data Analytics Courses in New Zealand

133. Fantastic blog. Learning programming languages like R and Python is difficult. However, I'm impressed with the manner you explained the underlying models. Gaining a deeper comprehension of the subject was made possible by the in-depth examination of variables and subsets. Thank you for the article, and please continue to share tech-related content. Digital marketing courses in patna

134. It's an excellent post. Great content on "DataBlog." Reading this article was fascinating. Your concise explanation of the Bootstrap Method and the General Linear Model is magnificent. I appreciate the effort you have put into this blog. So, I have learned a lot. Do post more. Financial modelling course in Singapore

135. I want to express my gratitude to you for establishing such a fantastic site. I also appreciate the time and effort you put into it, as well as the excellent explanation you provided. Congratulations and thanks for keeping us inspired. Show off more of your incredible work.
financial modelling course in bangalore

136. Excellent blog on tree modles. The points listed down are easy to understand. This kind of explanations will surely be useful to learners who want learning more about the topic. Amazing article. Keep posting more blogs.Shoring and Sheet Piling Contractors in Chennai

137. Hello Blogger,
you did a great one. Your post is concise and great to read. Thanks for making the post.
Data Analytics Courses in Zurich

138. Amazing article is about various sources of model uncertainty and how to deal with them. I am glad to find this article. Keep updating financial modelling course in gurgaon

139. Great article. The concept of " model uncertainty" is explained in-depth in this blog. The detailed description of "uncertainty inherent in the data and the algorithm" is simplified to the point. As a newbie, I found this tree model post valuable. I appreciate the effort and time put into this article. Thanks, and keep posting more helpful content. Data Analytics courses in leeds

data Analytics courses in liverpool

141. The article is very well described with diagrams and notes. Also liked the concept of the contents model uncertainty and how to deal with them. Also, if anyone is interested in learning more about Financial modelling course in Jaipur, then I would like to recommend you with this article on: financial modelling course in jaipur

142. Hello Pearson, I just want to thank you for this wonderful blog post. It is a fantastic one, and I was glad to go through it. Thank you for your great work. Data analytics fee in India can mean this Data Analytics Course Fee

143. Fantastic article.  This blog describes the idea of "model uncertainty" well. The thorough explanation of the "inherent uncertainty in the data and the algorithm" is condensed to the essentials. I appreciated this post on the tree model as a novice. I value the time and effort put into this article. Thank you, and do keep sharing helpful information. Data Analytics Scope

144. This post is excellent. The "model uncertainty" query is posed, and the response is well-explained. A concise explanation of inheriting some ambiguous data is provided. I learned a lot from this article as a novice. I appreciate all the effort that went into creating this post. The given formula is also intriguing. Thank you for posting more informative content. Continue to share excellent blogs in the future. Data Analyst Course Syllabus

145. This is a nice article here with some useful tips for those who are not used-to comment that frequently..Digital Marketing Agency in Pimple Saudagar

146. Hello blogger,
I think your blog post is a fantastic one. After I read, I found the content informative enough to share. Thanks again. Business Analytics courses in Pune

147. It's very nice to read article like this with details  Data Analyst Interview Questions

148. Hi dear blogger,
I really appreciate this article. In fact, I found the content so great, and like your ideas. thanks for all.
Data Analytics Qualifications

149. Hello, this blog on model uncertainty is very descriptive. I had no idea about this topic before but after reading this I feel I know a lot. Thank you for sharing this informative article.
Data Analytics VS Data Science

150. The article's concept on model uncertainty was well explained briefly in simple terms to understand. Also, if anyone is interested in learning more about Data Analyst Salary In India, then I would like to recommend you with this article to know and learn more about: Data Analyst Salary In India

151. I think your blog is fantastic. It has so many things to learn. Thanks for sharing.
CA Coaching in Mumbai

152. Hello blogger,
It is quite interesting to find blog post here. You a great way to use illustrations in it. I enjoyed reading this post.
Best Business Accounting & Taxation Course in India

153. Very knowledgeable topic about Model Uncertainty.. This blog has lot to learn Best Financial modeling courses in India

154. Hi dear blogger,
I was really excited to read this blog post. I found it interesting enough to read. A valid link for best SEO courses in India is this, Best SEO Courses in India

155. It is a very helpful and informative blog post.

156. This blog perfectly describes the idea of "model uncertainty. I appreciate the time and effort put into this article. keep sharing helpful information Best GST Courses in India

157. Hello blogger,
I just mean to thank you for this wonderful work you have done here. In fact, I am really please to find this blog post on this platform. I enjoyed reading it. Best Content Writing Courses in India

158. The fact that this article is well described with diagrams and notes on modern uncertainty was much useful for me. Also, if anyone is interested in learning more about Best GST Courses in India, then I would like to recommend you with this article on the Best GST Courses in India – A Detailed Exposition With Live Training. Best GST Courses in India

159. You have done a great job in discussing the issue of model uncertainty. You have presented different scenarios to highlight how the decisions made by data scientists are impacted by the model uncertainty. The examples you have used are very relevant and easy to understand. It is commendable that you have discussed multiple solutions to address this issue. Overall, you have presented an informative blog which is thought-provoking and insightful. Thank you for sharing your ideas and knowledge. Best Technical Writing Courses in India

160. You have written an excellent blog on model uncertainty. The topic of model uncertainty is difficult to explain in a simple way, but you have been successful in doing so. Your examples of measuring uncertainty in the form of confidence intervals and hypothesis testing are very informative and interesting. You have also provided practical advice on how to deal with uncertainty in the form of testing different models and picking the most appropriate one. Your insight into this subject is much appreciated. Thank you for making this blog so informative and enjoyable to read. FMVA

161. Hi very informative and well written. Got to know a lot. You have explained it precisely. Thanks a lot for sharing this information.
Data Analytics Courses in Kenya

162. Great job on this blog, you! It was really interesting to read about model uncertainty and the impact it can have on our results. You described the issue in a way that was easy to understand, and you provided a great example to illustrate the importance of understanding this concept. I'm sure this will be a huge help to many people who are just starting out in data analysis. Thanks for your hard work. Article Writing

163. Thank you for sharing such a nice and interesting blog with us.
Best Tally Courses in India

164. Informative blog! Thanks for sharing us. Best Business Accounting & Taxation Course in India

165. Thank you for writing this blog. It was really interesting to read about how uncertainty affects our models. You did a great job of covering the different types of uncertainties that can arise when dealing with models and how to tackle them. I particularly liked the idea of using different methods to quantify uncertainty and the importance of assessing the uncertainty of models. I appreciate your in-depth analysis of the topic and the insight you provide. Thank you for your hard work. Best Technical Writing Courses in India

166. Your blog post on "A Question of Model Uncertainty" is really well written. You have presented the concept of model uncertainty in a very clear and concise manner. You have discussed the importance of having a good understanding of the data in order to produce a reliable model. Your discussion about the tools used to measure the uncertainty of a model is quite impressive. You have discussed the importance of using these tools and how they can help to improve the model. Great job. Digital Marketing Courses in Glassglow