This blog is a companion to my recent book, Exploring Data in Engineering, the Sciences, and Medicine, published by Oxford University Press. The blog expands on topics discussed in the book, and the content is heavily example-based, making extensive use of the open-source statistical software package R.

Sunday, March 9, 2014

A question of model uncertainty

It has been several months since my last post on classification tree models, because two things have been consuming all of my spare time.  The first is that I taught a night class for the University of Connecticut’s Graduate School of Business, introducing R to students with little or no prior exposure to either R or programming.  My hope is that the students learned something useful – I can say with certainty that I did – but preparing for the class and teaching it took a lot of time.  The other activity, that has taken essentially all of my time since the class ended, is the completion of a book on nonlinear digital filtering using Python, joint work with my colleague Moncef Gabbouj of the Tampere University of Technology in Tampere, Finland.  I will have more to say about both of these activities in the future, but for now I wanted to respond to a question raised about my last post.

Specifically, Professor Frank Harrell, the developer of the extremely useful Hmisc package, asked the following:

            How did you take into account model uncertainty?  The uncertainty resulting from data mining to find nodes and thresholds for continuous predictors has a massive impact on confidence intervals for estimates from recursive partitioning.

The short answer is that model uncertainty was not accounted for in the results I presented last time, primarily because – as Professor Harrell’s comments indicate – this is a complicated issue for tree-based models.  The primary objective of this post and the next few is to discuss this issue.

So first, what exactly is model uncertainty?  Any time we fit an empirical model to data, the results we obtain inherit some of the uncertainty present in the data.  For the specific example of linear regression models, the magnitude of this uncertainty is partially characterized by the standard errors included in the results returned by R’s summary() function.  This magnitude depends on both the uncertainty inherent in the data and the algorithm we use to fit the model.  Sometimes – and classification tree models are a case in point – this uncertainty is not restricted to variations in the values of a fixed set of parameters, but it can manifest itself in substantial structural variations.  That is, if we fit classification tree models to two similar but not identical datasets, the results may differ in the number of terminal nodes, the depths of these terminal nodes, the variables that determine the path to each one, and the values of these variables that determine the split at each intermediate node.  This is the issue Professor Harrell raised in his comments, and the primary point of this post is to present some simple examples to illustrate its nature and severity.

In addition, this post has two other objectives.  The first is to make amends for a very bad practice demonstrated in my last two posts.  Specifically, the classification tree models described there were fit to a relatively large dataset and then evaluated with respect to that same dataset.  This is bad practice because it can lead to overfitting, a problem that I will discuss in detail in my next post.  (For a simple example that illustrates this problem, see the discussion in Section 1.5.3 of Exploring Data in Engineering, the Sciences, and Medicine.)  In the machine learning community, this issue is typically addressed by splitting the original dataset randomly into three parts: a training subset (Tr) used for model-fitting, a validation subset (V) used for intermediate modeling decisions (e.g., which variables to include in the model), and a test subset (Te) used for final model evaluation.  This approach is described in Section 7.2 of The Elements of Statistical Learning by Hastie, Tibshirani, and Friedman, who suggest 50% training, 25% validation, and 25% test as a typical choice.

The other point of this post is to say something about the different roles of model uncertainty and data uncertainty in the practice of predictive modeling.  I will say a little more at the end, but whether we are considering business applications like predicting customer behavior or industrial process control applications to predict the influence of changes in control valve settings, the basic predictive modeling process consists of three steps: build a prediction model; fix (i.e., “finalize”) this model; and apply it to generate predictions from data not seen in the model-building process.  In these applications, model uncertainty plays an important role in the model development process, but once we have fixed the model, we have eliminated this uncertainty by fiat.  Uncertainty remains an important issue in these applications, but the source of this uncertainty is in the data from which the model generates its predictions and not in the model itself once we have fixed it.  Conversely, as George Box famously said, “all models are wrong, but some are useful,” and this point is crucial here: if the model uncertainty is great enough, it may be difficult or impossible to select a fixed model that is good enough to be useful in practice.



Returning to the topic of uncertainty in tree-based models, the above plot is a graphical representation of a classification tree model repeated from my previous two posts.  This model was fit using the ctree procedure in the R package party, taking all optional parameters at their default values.  As before, the dataset used to generate this model was the Australian vehicle insurance dataset car.csv, obtained from the website associated with the book Generalized Linear Models for Insurance Data, by Piet de Jong and Gillian Z. Heller.  This model – and all of the others considered in this post – was fit using the same formula as before:

            Fmla = clm ~ veh_value + veh_body + veh_age + gender + area + agecat

Each record in this dataset describes a single-vehicle, single-driver insurance policy, and clm is a binary response variable taking the value 1 if policy filed one or more claims during the observation period and 0 otherwise.  The other variables (on the right side of “~”) represent covariates that are either numeric (veh_value, the value of the vehicle) or categorical (all other variables, representing the vehicle body type, its age, the gender of the driver, the region where the vehicle is driven, and the driver’s age).

As I noted above, this model was fit to the entire dataset, a practice that is to be discouraged since it does not leave independent datasets of similar character for validation and testing.  To address this problem, I randomly partitioned the original dataset into a 50% training subset, a 25% validation subset, and a 25% test subset as suggested by Hastie, Tibshirani and Friedman.  The plot shown below represents the ctree model we obtain using exactly the same fitting procedure as before, but applied to the 50% random training subset instead of the complete dataset.  Comparing these plots reveals substantial differences in the overall structure of the trees we obtain, strictly as a function of the data used to fit the models.  In particular, while the original model has seven terminal nodes (i.e., the tree assigns every record to one of seven “leaves”), the model obtained from the training data subset has only four.  Also, note that the branches in the original tree model are determined by the three variables agecat, veh_body, and veh_value, while the branches in the model built from the training subset are determined by the two variables agecat and veh_value only.



These differences illustrate the point noted above about the strong dependence of classification tree model structure on the data used in model-building.  One could object that since the two datasets used here differ by a factor of two in size, the comparison isn’t exactly “apples-to-apples.”  To see that this is not really the issue, consider the following two cases, based on the idea of bootstrap resampling.  I won’t attempt a detailed discussion of the bootstrap approach here, but the basic idea is to assess the effects of data variability on a computational procedure by applying that procedure to multiple datasets, each obtained by sampling with replacement from a single source dataset.  (For a comprehensive discussion of the bootstrap and some of its many applications, refer to the book Bootstrap Methods and their Application by A.C. Davison and D.V. Hinkley.)  The essential motivation is that these datasets – called bootstrap resamples – all have the same essential statistical character as the original dataset.  Thus, by comparing the results obtained from different bootstrap resamples, we can assess the variability in results for which exact statistical characterizations are either unknown or impractical to compute.  Here, I use this idea to obtain datasets that should address the “apples-to-apples” concern raised above.  More specifically, I start with the training data subset used to generate the model described in the previous figure, and I use R’s built-in sample() function to sample the rows of this dataframe with replacement.  For an arbitrary dataframe DF, the code to do this is simple:

> set.seed(iseed) 
> BootstrapIndex = sample(seq(1,nrow(DF),1),size=nrow(DF),replace=TRUE 
> ResampleFrame = DF[BootstrapIndex,]

The only variable in this procedure is the seed for the random sampling function sample(), which I have denoted as iseed.  The extremely complicated figure below shows the ctree model obtained using the bootstrap resample generated from the training subset with iseed = 5.





Comparing this model with the previous one – both built from datasets of the same size, with the same general data characteristics – we see that the differences are even more dramatic than those between the original model (built from the complete dataset) and the second one (built from the training subset).  Specifically, while the training subset model has four terminal nodes, determined by two variables, the bootstrap subsample model uses all six of the variables included in the model formula, yielding a tree with 16 terminal nodes.  But wait – sampling with replacement generates a significant number of duplicated records (for large datasets, each bootstrap resample contains approximately 63.2% of the original data values, meaning that the other 36.8% of the resample values must be duplicates).  Could this be the reason the results are so different?  The following example shows that this is not the issue.



This plot shows the ctree model obtained from another bootstrap resample of the training data subset, obtained by specifying iseed = 6 instead of iseed = 5.  This second bootstrap resample tree is much simpler, with only 7 terminal nodes instead of 16, and the branches of the tree are based on only four of the prediction variables instead of all six (specifically, neither gender nor veh_body appear in this model).  While I don’t include all of the corresponding plots, I have also constructed and compared the ctree models obtained from the bootstrap resamples generated for all iseed values between 1 and 8, giving final models involving between four and six variables, with between 7 and 16 terminal nodes.  In all cases, the datasets used in building these models were exactly the same size and had the same statistical character.  The key point is that, as Professor Harrell noted in his comments, the structural variability of these classification tree models across similar datasets is substantial.  In fact, this variability of individual tree-based models was one of the key motivations for developing the random forest method, which achieves substantially reduced model uncertainty by averaging over many randomly generated trees.  Unfortunately, the price we pay for this improved model stability is a complete loss of interpretibility.  That is, looking at any one of the plots shown here, we can construct a simple description (e.g., node 12 in the above figure represents older drivers – agecat > 4 – with less expensive cars, and it has the lowest risk of any of the groups identified there).  While we may obtain less variable predictions by averaging over a large number of these trees, such simple intuitive explanations of the resulting model are no longer possible.

I noted earlier that predictive modeling applications typically involve a three-step strategy: fit the model, fix the model, and apply the model.  I also argued that once we fix the model, we have eliminated model uncertainty when we apply it to new data.  Unfortunately, if the inherent model uncertainty is large, as in the examples presented here, this greatly complicates the “fix the model” step.  That is, if small variations in our training data subset can cause large changes in the structure of our prediction model, it is likely that very different models will exhibit similar performance when applied to our validation data subset.  How, then, do we choose?  I will examine this issue further in my next post when I discuss overfitting and the training/validation/test split in more detail. 


291 comments:

  1. I am trying to use CART algorithms as against conventional statistics, DOE and hypothesis testing. I have got some results and advantages also for problem solving rather than using the data as training set for future prediction.

    Is my understanding correct that uncertainty is more important in cases where we use this training data rather than problem solving approach?

    ReplyDelete

  2. It looks like you spend a lot of effort and time on your blog. I have bookmarked it and I am looking forward to reading new articles. Keep up the good work
    Android Training in Chennai

    ReplyDelete
  3. I am expecting more interesting topics from you. And this was nice content and definitely it will be useful for many people.

    Hadoop Training in Chennai

    ReplyDelete
  4. Wow it is really wonderful and awesome thus it is very much useful for me to understand many concepts and helped me a lot.thus these tips are really awesome and you had a wonderful products.


    Online Reputation Management

    ReplyDelete
  5. great tips thanks for your impressive enhancement of getting unique style of design a website. and it much more wonderful information to me. keep share many different ideas.
    IELTS coaching in chennai

    ReplyDelete
  6. Truly a very good article on how to handle the future technology. After reading your post,thanks for taking the time to discuss this, I feel happy about and I love learning more about this topic
    Tooth Braces In Chennai

    ReplyDelete
  7. wonderful post and very helpful, thanks for all this information. You are including better information regarding this topic in an effective way.Thank you so much
    Web Design Company in Chennai


    ReplyDelete
  8. Thank you for sharing such a nice and interesting blog with us. But in your blog, I had a chance to get some useful and unique information. I would like to suggest your blog.
    Web development Company in Chennai

    ReplyDelete
  9. this is very nice blog this studying course information very useful to everyone who have learning this information.

    Base SAS Training in chennai

    ReplyDelete
  10. This is a nice article here with some useful tips for those who are not used-to comment that frequently.Thanks for this helpful information I agree with all points you have given to us.I will follow all of them.

    Self Employment Tax
    Tax Preparation Services
    Tax Accountant
    Tax Consultant
    Tax Advisor
    Online Tax services
    Tax Professional
    Online Tax Preparation"

    ReplyDelete
  11. These ways are very simple and very much useful, as a beginner level these helped me a lot thanks fore sharing these kinds of useful and knowledgeable information.

    Hadoop Training in Chennai

    ReplyDelete
  12. useful and knowledgeable information.
    thank u sharing!!


    This blog was very useful for me waiting for more blog.


    SAP FICO Training in Chennai

    ReplyDelete
  13. It is really very excellent. Because of all given information was wonderful and it's very helpful for me. Thanks for sharing
    SAP Training in Chennai
    SAP ABAP Training in Chennai
    SAP FICO Training in Chennai
    SAP MM Training in Chennai

    ReplyDelete
  14. These ways are very simple and very much useful, as a beginner level these helped me a lot thanks fore sharing these kinds of useful and knowledgeable information.
    Also Check out the : https://www.credosystemz.com/training-in-chennai/best-data-science-training-in-chennai/

    ReplyDelete
  15. Benefits of reading your post. This will help in the maintenance of our health.The daily routine will help you lose weight quickly and safely.My life is totally transformed after I followed this diet.I feeling great about myself.

    Herbalife in Chennai
    Herbalife Nutrition Products
    Nutritionclub in Chennai
    Weight Loss in Chennai
    Weight Gain in Chennai

    ReplyDelete
  16. Your site is amazing and your blogs are informative and knlowledgeble to my websites.This is one of the best tips in my life.I have in quite some time.Nicely written and great info.Thanks to share the more informations.

    Seo Experts
    Seo Company
    Web Designing Company
    Digital Marketing
    Web Development Company
    Apps Development

    ReplyDelete
  17. This is a great post. I like this topic.This site has lots of advantage. I found many interesting things from this site. It helps me in many ways.Thanks for posting this again.
    Best ERP Software
    ERP Software In India
    ERP Companies In Chennai
    ERP Software In Chennai

    ReplyDelete
  18. Amazing article. Your blog helped me to improve myself in many ways thanks for sharing this kind of wonderful informative blogs in live. I have bookmarked more article from this website. Such a nice blog you are providing ! Kindly Visit Us

    R Programming institutes in Chennai | R Programming Training in Chennai | R Programming Course Fees | R Language training in Chennai

    ReplyDelete
  19. Great post! I am actually getting ready to across this information, It's very helpful for this blog.Also great with all of the valuable information you have Keep up the good work you are doing well. big data training in Velachery | Hadoop Training in Chennai | big data Hadoop training and certification in Chennai | Big data course fees |

    ReplyDelete
  20. I feel happy about and learning more about this topic. keep sharing your information regularly for my future reference. This content creates new hope and inspiration within me. Thanks for sharing an article like this. the information which you have provided is better than another blog.
    IELTS Coaching in Dwarka mor

    ReplyDelete
  21. Great Post,really it was very helpful for us.
    Thanks a lot for sharing!
    I found this blog to be very useful!!
    Hadoop training in Bangalore

    ReplyDelete
  22. Vanskeligheter( van bi ) vil passere. På samme måte som( van điện từ ) regnet utenfor( van giảm áp ) vinduet, hvor nostalgisk( van bướm ) er det som til slutt( van cửa ) vil fjerne( van công nghiệp ) himmelen.

    ReplyDelete
  23. Si el agua cae al lago, desaparecerá( phụ kiện tủ bếp ). Pero si cae a la hoja de( phụ kiện tủ áo ) loto, brillará como una joya. Caer igual pero( thùng gạo thông minh ) estar con alguien es importante.

    ReplyDelete
  24. Good blog article
    Sanjary Kids is one of the best play school and preschool in Hyderabad,India. Give your child the best preschool experience by choosing the best playschool of Hyderabad in Abids. we provide programs like Play group,Nursery,Junior KG,Senior KG,and provides Teacher Training Program.
    Preschool in hyderabad

    ReplyDelete
  25. Good blog posts information by the author

    Sanjary Academy is the best Piping Design institute in Hyderabad, Telangana. It is the best Piping design Course in India and we have offer professional Engineering Courses like Piping design Course, QA/QC Course, document controller course, Pressure Vessel Design Course, Welding Inspector Course, Quality Management Course and Safety Officer Course.
    Piping Design Course
    Piping Design Course in Hyderabad ­
    Piping Design Course in India­

    ReplyDelete
  26. Good blog information I liked it

    Sanjary Academy is the best Piping Design institute in Hyderabad, Telangana. It is the best Piping design Course in India and we have offer professional Engineering Courses like Piping design Course, QA/QC Course, document controller course, Pressure Vessel Design Course, Welding Inspector Course, Quality Management Course and Safety Officer Course.
    Piping Design Course
    Piping Design Course in Hyderabad ­
    Piping Design Course in India­

    ReplyDelete
  27. Great blog information and understandable

    Sanjary Academy provide pressure vessel design,quality management system course, piping design course, qa/qc course and document controller course.
    Welding Inspector Course
    Safety officer course
    Quality Management Course
    Quality Management Course in India

    ReplyDelete
  28. Thanks for sharing information
    "Sanjary Academy provides excellent training for Piping design course. Best Piping Design Training Institute in Hyderabad,
    Telangana. We have offer professional Engineering Course like Piping Design Course,QA / QC Course,document Controller
    course,pressure Vessel Design Course, Welding Inspector Course, Quality Management Course, #Safety officer course."
    Piping Design Course in India­
    Piping Design Course in Hyderabad
    Piping Design Course in Hyderabad
    QA / QC Course
    QA / QC Course in india
    QA / QC Course in Hyderabad
    Document Controller course
    Pressure Vessel Design Course
    Welding Inspector Course
    Quality Management Course
    Quality Management Course in india
    Safety officer course

    ReplyDelete
  29. Nice blog..! I really loved reading through this article. Thanks for sharing.You done a great job.Oracle Applications Training in Bangalore

    ReplyDelete
  30. Thank you for excellent article.You made an article that is interesting.
    Digital marketing course in Bangalore with Live Projects. 100% placement, 20+ modules, 10+ certifications, Great discounts on course fees.
    https://onlineidealab.com/digital-marketing-courses-in-bangalore/

    ReplyDelete
  31. Thanks for one marvelous posting! I enjoyed reading it; you are a great author. I will make sure to bookmark your blog and may come back someday. I want to encourage that you continue your great posts.






    learn digital marketing

    ReplyDelete
  32. Awesome article, it was exceptionally helpful! I simply began in this and I'm becoming more acquainted with it better! Cheers, keep doing awesome
    All University BCOM 1st Year TimeTable 2020

    ReplyDelete
  33. This was definitely one of my favorite blogs. Every post published did impress me. ExcelR Digital Marketing Class In Pune

    ReplyDelete
  34. I concur with a ton of the focuses you made in this article. I value the work you have placed into this and expectation you keep composing regarding this matter.
    Best Data Science training in Mumbai

    Data Science training in Mumbai

    ReplyDelete
  35. Thanks really it is very useful blog obia online training Evanta technologies provide you the best OBIA online training with step by step procedure for installing, customizing and configuring of OBIA.

    ReplyDelete
  36. I am glad to read that you come up with outstanding information that definitely allows me to share with others. Thank you for sharing this with us.

    digital marketing coaching in hubli

    ReplyDelete
  37. Your info is really amazing with impressive content..Excellent blog with informative concept. Really I feel happy to see this useful blog, Thanks for sharing such a nice blog..
    Android Training Institute in Chennai | Android Training Institute in anna nagar | Android Training Institute in omr | Android Training Institute in porur | Android Training Institute in tambaram | Android Training Institute in velachery


    ReplyDelete
  38. It is a very helpful and informative blog post. I would like to thank to you for providing such information I have also have a website providing very good information.good job ..keep it up
    Ai & Artificial Intelligence Course in Chennai
    PHP Training in Chennai
    Ethical Hacking Course in Chennai Blue Prism Training in Chennai
    UiPath Training in Chennai

    ReplyDelete
  39. Thanks for writing this article such a good...it helps to all people for our health related problems like muscle pain, chronic pain, or joint pain and so on...
    Click Here
    Buy soma 350mg online
    Order Soma 500mg cash on delivery in cheap price
    Buy Soma (Carisoprodol) Online COD
    Order Soma 500mg Online COD with PayPal and Credit/Debit Card

    ReplyDelete
  40. Know about Buying Tramadol 100mg Cash On Delivery. Get Tramadol In Best 2020 Price Online thru Cash On Delivery C.O.D option online.you can order Tramadol 100mg in USA cash on delivery facility.Order Tramadol Online thru Cash on delivery option. Doctors generally prescribe this medication for the treatment of moderate and moderately severe pain. Order Tramadol COD online now in cheapest price with fast and secure Cash on Delivery.

    To Cheap Tramadol cash on delivery Contact Us: 916-587-2469 and You Can Also E-mail: PharmaHelpline247@gmail.com.

    ReplyDelete
  41. Really amazing experience to visit your blog, thanks a lot for sharing with us. Visit Ogen Infosystem for creative website designing and ppc services at best price in Delhi.
    Web Designing Company

    ReplyDelete

  42. Hi
    I visited your blog you have shared amazing information, i really like the information provided by you, You have done a great work. I hope you will share some more information regarding full movies online. I appreciate your work.
    Thanks

    OpenStack Training in Bangalore

    ReplyDelete
  43. Thanks mate. I am really impressed with your writing talents and also with the layout on your weblog. Appreciate, Is this a paid subject matter or did you customize it yourself? Either way keep up the nice quality writing, it is rare to peer a nice weblog like this one nowadays. Thank you event marketing and How to Write a Speaker Biography for a Conference

    ReplyDelete

  44. Very interesting article to read it. I would like to thank you for the efforts you had made for writing this wonderful article. This article inspired me to read more. Keep sharing on updated posts…

    Learn Digital Marketing Course in Bangalore with Live Project Work & Case Studies taught by Ranjan Jena (10Yrs Trainer). 100% Guarantee to Clear Job Interview

    ReplyDelete
  45. BUY SOMA MEDS

    Buy Adderall Online
    Buy Alprazolam Online
    Buy Ambien Online
    Buy Ativan Online
    Buy Butalbital Online
    Buy Carisoprodol Online
    Buy Fioricet Online
    Buy Klonopin Online
    Buy Soma Online
    Buy Tramadol Online
    Buy Ultram Online
    Buy Valium Online
    Buy Xanax Online
    Green Xanax Bars [S 90 3]
    White Xanax Bars
    Yellow Xanax Bars

    ReplyDelete
  46. Your blog provided us with valuable information to work with. Each & every tips of your post are awesome. Thanks a lot for sharing. Keep blogging,
    Best Digital Marketing Institute in Hyderabad

    ReplyDelete
  47. So you can save your lot of time by hiring SEO Freelancer or if you want to learn full SEO Course Delhi

    ReplyDelete
  48. Thanks for the always useful information. This is great information to help peoples and nice article written by writer. CnX Player is a powerful & efficient 4K ultra HD enabled video player for Windows 10 PC & Tablet, Android and iOS – iPhone & iPad.

    Download Media Player for Windows 10 - Microsoft Store
    Download Video Player for Android from Google Play
    Download Video Player for iPhone/iPad from Apple App Store

    ReplyDelete
  49. Thanks for posting the best information and the blog is very helpfuldigital marketing institute in hyderabad

    ReplyDelete
  50. This site helps to clear your all query.
    This is really worth reading. nice informative article.
    uttar matric scholarship
    madhya pradesh post matric scholarship

    ReplyDelete
  51. Thanks for choosing this specific Topic. As i am also one of big lover of this. Your explanation in this context is amazing. Keep posted these overwarming facts in front of the world.
    Printer customer support number

    ReplyDelete
  52. Thanks for sharing.

    ERP HR Module in Bangalore
    Zenetial is the best HR module in ERP Bangalore. This ERP payroll software manages all your employee details from hire to retire. Get a free demo of HR software.

    ReplyDelete
  53. Thanks for sharing.
    Best Aeronautical Admissions in Bangalore
    Are you are looking for aeronautical engineering for studying in Bangalore. We can proudly say we are the best aeronautical admissions providers in top colleges and universities.

    ReplyDelete
  54. IELTS is known for the best IELTS coaching in Ahmedabad and consistently successful results. High Band Score 8 and Average Band Score Minimum 6.8 are maintained so that you can get more information about IELTS Coaching Institute, Ahmedabad.


    ILETS is known for the best quality ILETS coaching and consistently successful results. Maintaining the highest band score of 8 and the average band scores minimum of 6.8 to help you to learn more about the IELTS Institute of Ahmedabad

    ReplyDelete
  55. Amazing write-up! Really Good.
    Now the foundation of any business is marketing. Adsify marketing is the best Digital marketing in Trivandrum.

    ReplyDelete
  56. they are very simple and usefull information thanksfor sharing

    ReplyDelete
  57. Thanks for the blog but If your looking to switch your career to the Airline Industry we the Fusion is the best place to pursue. It is one of the best Cabin Crew Training Centre

    ReplyDelete
  58. Thank you for taking the time to provide us with your valuable information.
    Cheapest email marketing Services In India

    ReplyDelete
  59. Great! Thanks for sharing this kind of information it’s really very nice and useful for me.
    Website Designing Company in South Delhi
    Web Plus Era is a leading Website Designing Company in Dwarka. We offer web development solutions, but also Delhi web services. Mobile apps have been the center of attraction for most of the people. For almost all e-commerce websites and business operators, mobile apps are a wonderful way to make your way to a lot of major customers, clients, and a large market segment. Web Plus Era is the best App Development Company in Delhi. Responsive Website designs are in excessive call for because of their better functionalities and responsiveness to make stop person revel in a memorable one. Companies are making an investment large on responsive net layout offerings. Web Plus Era is among the quickest rising Responsive Web Design Services responsive net designing organizations within side the India,Delhi. Website Designing Company in DelhiVery satisfied with the Website designing work effort by your team! Happy to recommend it to anyone looking for Website Designing service Web Plus Era is best Place for your Digital Work like website,seo,smo etc.

    ReplyDelete
  60. Thanks for sharing such an informational blog which will surely be a big help to the small medium enterprise so that they can choose the best suited tool for their business.
    top 10 crm software in india

    ReplyDelete
  61. great post, I enjoy it. I also buy ambien a lot. You need to buy ambien as well. I will travel to the whole USA and buy ambien there and in other areas to. i will be downtown, you know for what ? just to buy ambien. I will sleep a ton immediately after buy ambien and that it usually make me feel so very good. I will be buy ambien in a pharmacy or a different locations, no matter what the existence consider me . Judi Online

    ReplyDelete
  62. Dead pent subject material , Really enjoyed examining . hotmail

    ReplyDelete
  63. Thank you for giving the great article. It delivered me to understand several things about this concept. Keep posting such surpassing articles so that I gain from your great post.
    ibm full form in india |
    ssb ka full form |
    what is the full form of dp |
    full form of brics |
    gnm nursing full form |
    full form of bce |
    full form of php |
    bhim full form |
    nota full form in india |
    apec full form

    ReplyDelete
  64. good info about model uncertainity
    B2B Business Leads

    ReplyDelete
  65. Appreciation for sharing this bend towards. I in truth other than your weblog uncover no think about packs. you have in truth shared an illuminating and drawing in weblog dissipate following individuals.. Free Quick Heal Product Key

    ReplyDelete
  66. i'm glad to find this say incredibly profitable for me, since it involves part of are searching for. I consistently choose to affirmation the man or woman content and this case I found in you announce. much gratitude to you for sharing. Wifi Password Hacker For Pc

    ReplyDelete
  67. Ive been attempting to find a few respectable stuff concerning the challenge and dock't had any good fortune going on till this narrowing, You simply got a subsidiary biggest lover!.. Best Farewell Quotes For Friends

    ReplyDelete
  68. Awesome article! You are providing us very useful information. Thank you for sharing. Check out Digital Marketing Institute in Pune

    ReplyDelete
  69. I am really grateful for giving me an opportunity to read a good content.
    Digital marketing courses in New zealand

    ReplyDelete
  70. This blog is a simple plus has a detailed analysis on the Search Engine Marketing. Which gives us the advantage plus the disadvantages of various SEM's. To know more do visit -
    Search Engine Marketing

    ReplyDelete
  71. Thank you for sharing your knowledge in this blog. It will help many people in exploring data area. Keep posting. Content Writing Course in Bangalore

    ReplyDelete
  72. Your article has to be one of the most detailed blogs out there for the kind of information provided, keep publishing more articles to spread more knowledge. And those who are searching for Digital Marketing Courses in Nigeria can refer the following blog, thank you.

    Digital marketing courses in Nigeria

    ReplyDelete
  73. Nice and detailed post. Looking to learn digital marketing in Dehradun with hands on training by the industry experts then visit us: Digital Marketing Course in Dehradun

    ReplyDelete
  74. I was looking for this exact info and thankfully I came across this article. Digital marketing courses in Ahmedabad

    ReplyDelete
  75. A very descriptive content with infographics and illustrated manner. Explained very well on the topic. Thanks for sharing. If someone is looking for Digital Marketing Course in France then follow the link and go through to get the entire details of the course and other courses as well. This is the full power-packed course content you can acquire great expertise by joining our comprehensive course.
    Digital marketing courses in france

    ReplyDelete
  76. Thank you for the information
    I concur that this is the most thorough description of the subject. I'm so glad I found your blog and am looking forward to reading your future posts. And I have referred to related content in the link below.

    At Login360, you may get the best training in android Training in Chennai
    . We provide a variety of software-related courses along with complete placement assistance.

    Excellent IT instruction has been given to our pupils in a number of methods by our teachers and subject-matter specialists.

    We offer top-notch instruction in Android technologies, and we frequently update our curricula to include the most recent IT trends.

    We provide placement help for recent grads (recent graduates). We will offer support to all eligible applicants.

    Contact Details:
    Name: Login360 Software Training Institute
    Address: No-06, Ground Floor, 5th Main Road, Vijaya Nagar Velachery, Chennai – 600042.
    Phone: 6385872810

    ReplyDelete
  77. The article is well informed with logical ideas and also it is helpful as well. Digital marketing courses in Agra

    ReplyDelete
  78. Wow, you have written very informative content. Looking forward to reading all your blogs. If you want to read about Online SOP please click Online SOP

    ReplyDelete
  79. Great informative blog. This is a valuable source of information. Thanks for sharing your knowledge over here. Please check the Digital Marketing Courses in Delhi to know more about the topic. Surely you already know the power of Digital Marketing when you want to boost your website, your career or your business. Read more here: Digital Marketing Courses in Delhi

    ReplyDelete
  80. This is by far one of the most engaging articles I have read in recent times. Just loved the quality of information provided and I must say you have noted down the points very precisely, keep posting more. Digital Marketing is now booming at a rapid pace, especially in Dubai, and many are now searching for the courses. So to ease their work I am leaving a link below for those who are searching for Digital Marketing courses in Abu Dhabi. All the best and keep learning, thank you.
    Digital Marketing Courses in Abu Dhabi

    ReplyDelete
  81. Excellent blog. The time and efforts you have put to make this can be clearly seen. Thank you for sharing this with us.

    Looking to learn about digital marketing in nagpur then check out this informative article Digital marketing courses in Nagpur it provides all the valuable information you need.

    ReplyDelete
  82. Highly appreciable content!
    If anyone is interested in building a medical career but are struggling to clear medical entrance exams, Wisdom Academy is the right place to begin. It is one of Mumbai’s best NEET coaching institutes for students preparing for medical and other competitive-level entrance examinations. It offers comprehensive learning resources, advanced study apparatus, doubt-clearing sessions, mentoring, and much more. Enroll Now!
    NEET Coaching in Mumbai

    ReplyDelete
  83. This blog is very informative and helps many readers learn more. Thank you for posting.
    Digital marketing courses in Noida

    ReplyDelete
  84. I read your article and I really appreciate the fact you have given so much effort in sharing with us this good knowledge. Keep growing and keep sharing with us such lovely contents.
    Also do visit to our Digital Marketing courses in Bahamas

    ReplyDelete
  85. Good Works! Its a great pleasure reading your post.Its full of information I am looking for and I love to post a comment that “The content of your post is awesome” Great work. Content Writing Courses in Delhi

    ReplyDelete
  86. Worth reading article. This is a nice post for those who are interested in Data in Engineering. Thanks for sharing it here. We also provide an informational and educational blog about Freelancing. Nowadays, many people want to start a Freelance Career without knowing How and Where to start. People are asking:
    What is Freelancing and How Does it work?
    How to Become a Freelancer?
    Is working as a Freelancer a good Career?
    How much can a Freelancer earn?
    Can I live with a Self-Employed Home Loan?
    What Kind of Freelancing Jobs can I find?
    Which Freelancers Skills are required?
    How to get Freelance projects?
    How Do companies hire Freelancers?
    In our Blog, you will find a guide with Tips and Steps which will help you to take a good decision. Start reading and find out the Answers:
    What is Freelancing

    ReplyDelete
  87. Nice article . Thank you for this beautiful content, Keep it up. This blog has clear about related to the model uncertainty with detailed information. Thanks again for sharing. Financial Modeling Courses in India

    ReplyDelete
  88. Very well explained! Keep posting more contents.
    If you are searching for the ideal IAS coaching institute in Hyderabad, there is no better choice than Pragnya IAS Academy. It is one of the best coaching institutes in Hyderabad, offering civil services coaching.
    Anchor ias coaching in Hyderabad

    ReplyDelete
  89. model uncertainty is described with smart content and explanation is extremely wonderful. Its really my luck to have come across this blog. Its really a great content. Digital marketing courses in Kota

    ReplyDelete
  90. Great effort makes me understand the concept better.Thanks for sharing.
    Are you looking for the best financial modeling courses in India? Financial modeling skills are increasingly important to upgrade your career and work in the ever-growing finance sector. This article lists the best colleges in India that provide financial modeling courses. Continue reading to choose the best course for you.
    Financial Modeling Courses in India

    ReplyDelete
  91. About the article, I just wanted to say it is really nice and informative. Digital Marketing Courses in Faridabad

    ReplyDelete
  92. Truly great informative article on R. This most sought language and package is used most widely now for data mining and big analysis and data engineering. The article is very well described with diagrams and notes. Great work. Thanks for sharing. Keep sharing. I truly enjoyed the content by knowing deep insight and issues. If anyone wants to build his carrier in Digital Marketing then you must go through our curriculum which is designed very professionally with cutting edge of the current requirement of the corporates and based on market trends. You will be taught in a highly professional environment with practical assignments. You can decide your specialized stream your own way by understanding each subject in depth under the guidance of highly professional and experienced trainers. For more detail Please visit at
    Digital Marketing Courses in Austria

    ReplyDelete
  93. What an splendid and articulate blog. Enjoyed reading it. Do share more articles. Thanks.

    Digital marketing courses in Chennai

    ReplyDelete
  94. Truly a very good service with add-on features to make available the content to broader audience. Thanks for sharing the update on A question of model uncertainty. If anyone wants to learn Digital Marketing, please join the world-class curriculum and industry-standard professional skills. For more details, please visit Digital marketing Courses in Bhutan

    ReplyDelete
  95. Awesome article on uncertainty function and predictive modeling, very well explained.
    Digital marketing courses in Raipur

    ReplyDelete
  96. I really appreciate your efforts in creating this incredible article, thank you for taking the time to explain the topic in detail. It's authors like you who encourage readers to keep reading and to learn more.
    Data Analytics Courses In Kolkata

    ReplyDelete
  97. Your articles touch differently. I love your writing so much. keep share more article in future also. thank you so much.If someone is looking for data analytics courses in Indore then here is top 5 courses explained in this blog. Please check once for more information. Data Analytics Courses In Indore

    ReplyDelete
  98. very good article on uncertainty model. Thanks for sharing
    Data Analytics Courses In Ahmedabad

    ReplyDelete
  99. This blog post discusses the different sources of model uncertainty and how they can be addressed. It is a well-written and informative post that will be helpful for those looking to learn more about this topic. Data Analytics Courses In Coimbatore

    ReplyDelete
  100. Hi, I have gone through few of your blogs and must say they are really interesting as well as helpful. This blog clearly shows the efforts put in by the blogger to come up with the excellent in detail algorithms. Looking forward to more of such blogs.
    Data Analytics Courses In Kochi

    ReplyDelete
  101. Great topic about A question of model uncertainty. This post does a great job of outlining the different types of model uncertainty, and how they can impact the results of data analysis. Thanks for sharing with us Digital Marketing Courses in Australia

    ReplyDelete
  102. The article is worth reading every bit of it and gives a learning experience from this. Also, keep us updated about the post. Data Analytics Courses in Delhi

    ReplyDelete
  103. It's an excellent post. Nice blog on DataBlog. Fascinating to read this article. The way you have explained it is simple and understandable. The clear-cut description of General Linear Model and Bootstrap Methods. Thank you for the concise explanation and beneficial information. Well, I have gained a lot of understanding. Keep sharing more.
    Digital marketing courses in Nagpur

    ReplyDelete
  104. Mind boggling blog post, the writer has done a great job in explaining the model uncertainty. Kudos to the writer! Data Analytics Courses In Coimbatore , This course is designed for students who want to pursue a career in data analytics or for those who want to gain a better understanding of how data analytics can be used to improve business decision making.

    ReplyDelete
  105. This is a very unique blog on the topic of A question of model uncertainty. I really liked reading and is one of the greatest read for me. Thanks for sharing this blog with us. Keep up the good work. Digital Marketing Courses in Vancouver

    ReplyDelete
  106. The various sources of model uncertainty are covered in this blog post, along with solutions. Those who want to learn more about this subject will find this to be a well-written and instructive post.
    Data Analytics Courses in Ghana

    ReplyDelete
  107. This post is excellent. I like this information. The benefits and negatives of this website are numerous. This blog post contained a lot of valuable information. It benefits me greatly. I appreciate you reposting this.
    Data Analytics Courses in Mumbai

    ReplyDelete
  108. This exploring data blog is just amazing that how data used in various fields. I am looking for more data blogs. Keep posting.  Data Analytics Courses In Vadodara 

    ReplyDelete
  109. There's a lot to learn from this blog. Keep posting more blogs about data Digital marketing courses in Varanasi

    ReplyDelete
  110. Thanks for sharing the importance of data in the field of science, engineering, medicine. I would like to read more of this blog  Data Analytics Courses in navi Mumbai 

    ReplyDelete
  111. Your blogs are simple & easy to understand. I really enjoyed reading this.  Data Analytics Courses In Bangalore 

    ReplyDelete
  112. What a beautiful piece of writing! This is among the greatest articles I've seen regarding the model of uncertainty so far. Congratulations to the blogger for such a fantastic job! We are very grateful that you shared this important information with us.
    Data Analytics Courses in Gurgaon

    ReplyDelete
  113. Nicely written, good blog. Keep sharing valuable content for references.
    Data Analytics Courses in Visakhapatnam

    ReplyDelete
  114. Amazing blog. Learning programs such as R and Python is a challenging thing. But the way you have explained the base models is impressive. The in-depth discussion on variables and subsets helped to gain more understanding of the topic. Thanks for the article, and keep sharing more tech-oriented posts in the future.
    Courses after bcom

    ReplyDelete
  115. Awesome blog. I enjoyed reading your article. I would like to thank you for sharing your content and ideas with yours. Keep up the good work.If your are looking for professional courses after senior secondary here is the list of top 20 professional courses please check on this link - Professional Courses

    ReplyDelete
  116. Great article very informative on Data Tree model of R language which is very effective and informative and described in a very descriptive manner with scripts and narratives. If anyone wants to build his carrier in Digital Marketing then you must go through our curriculum which is designed very professionally with cutting edge of the current requirement of the corporates and based on market trends. For more detail Please visit at
    Digital marketing Courses In UAE

    ReplyDelete
  117. This fantastic blog explores the various sources of model uncertainty and how to deal with them. Those who want to learn more about this subject will find this to be a well-written and instructive post.
    financial modelling course in kenya

    ReplyDelete
  118. I like the fact what you have acquired here like what you are stating and the way in which you have said it. You make it enjoyable and I have bookmarked it as well. Also, I am anticipating in perusing similar new articles. Data Analytics Courses in New Zealand

    ReplyDelete
  119. Fantastic blog. Learning programming languages like R and Python is difficult. However, I'm impressed with the manner you explained the underlying models. Gaining a deeper comprehension of the subject was made possible by the in-depth examination of variables and subsets. Thank you for the article, and please continue to share tech-related content. Digital marketing courses in patna

    ReplyDelete
  120. It's an excellent post. Great content on "DataBlog." Reading this article was fascinating. Your concise explanation of the Bootstrap Method and the General Linear Model is magnificent. I appreciate the effort you have put into this blog. So, I have learned a lot. Do post more. Financial modelling course in Singapore

    ReplyDelete
  121. I want to express my gratitude to you for establishing such a fantastic site. I also appreciate the time and effort you put into it, as well as the excellent explanation you provided. Congratulations and thanks for keeping us inspired. Show off more of your incredible work.
    financial modelling course in bangalore

    ReplyDelete
  122. Excellent blog on tree modles. The points listed down are easy to understand. This kind of explanations will surely be useful to learners who want learning more about the topic. Amazing article. Keep posting more blogs.Shoring and Sheet Piling Contractors in Chennai

    ReplyDelete
  123. Hello Blogger,
    you did a great one. Your post is concise and great to read. Thanks for making the post.
    Data Analytics Courses in Zurich

    ReplyDelete
  124. Amazing article is about various sources of model uncertainty and how to deal with them. I am glad to find this article. Keep updating financial modelling course in gurgaon

    ReplyDelete
  125. Great article. The concept of " model uncertainty" is explained in-depth in this blog. The detailed description of "uncertainty inherent in the data and the algorithm" is simplified to the point. As a newbie, I found this tree model post valuable. I appreciate the effort and time put into this article. Thanks, and keep posting more helpful content. Data Analytics courses in leeds

    ReplyDelete
  126. What a helpful article about the concept of Model Uncertainty. This article was quite interesting to read. I want to express my appreciation for your time and making this fantastic post.
    data Analytics courses in liverpool

    ReplyDelete
  127. The article is very well described with diagrams and notes. Also liked the concept of the contents model uncertainty and how to deal with them. Also, if anyone is interested in learning more about Financial modelling course in Jaipur, then I would like to recommend you with this article on: financial modelling course in jaipur

    ReplyDelete
  128. Hello Pearson, I just want to thank you for this wonderful blog post. It is a fantastic one, and I was glad to go through it. Thank you for your great work. Data analytics fee in India can mean this Data Analytics Course Fee

    ReplyDelete
  129. Fantastic article.  This blog describes the idea of "model uncertainty" well. The thorough explanation of the "inherent uncertainty in the data and the algorithm" is condensed to the essentials. I appreciated this post on the tree model as a novice. I value the time and effort put into this article. Thank you, and do keep sharing helpful information. Data Analytics Scope

    ReplyDelete
  130. This post is excellent. The "model uncertainty" query is posed, and the response is well-explained. A concise explanation of inheriting some ambiguous data is provided. I learned a lot from this article as a novice. I appreciate all the effort that went into creating this post. The given formula is also intriguing. Thank you for posting more informative content. Continue to share excellent blogs in the future. Data Analyst Course Syllabus

    ReplyDelete
  131. This is a nice article here with some useful tips for those who are not used-to comment that frequently..Digital Marketing Agency in Pimple Saudagar

    ReplyDelete
  132. Hello blogger,
    I think your blog post is a fantastic one. After I read, I found the content informative enough to share. Thanks again. Business Analytics courses in Pune

    ReplyDelete
  133. Hi dear blogger,
    I really appreciate this article. In fact, I found the content so great, and like your ideas. thanks for all.
    Data Analytics Qualifications

    ReplyDelete
  134. Hello, this blog on model uncertainty is very descriptive. I had no idea about this topic before but after reading this I feel I know a lot. Thank you for sharing this informative article.
    Data Analytics VS Data Science

    ReplyDelete
  135. The article's concept on model uncertainty was well explained briefly in simple terms to understand. Also, if anyone is interested in learning more about Data Analyst Salary In India, then I would like to recommend you with this article to know and learn more about: Data Analyst Salary In India

    ReplyDelete
  136. I think your blog is fantastic. It has so many things to learn. Thanks for sharing.
    CA Coaching in Mumbai

    ReplyDelete
  137. Hello blogger,
    It is quite interesting to find blog post here. You a great way to use illustrations in it. I enjoyed reading this post.
    Best Business Accounting & Taxation Course in India

    ReplyDelete
  138. Very knowledgeable topic about Model Uncertainty.. This blog has lot to learn Best Financial modeling courses in India

    ReplyDelete
  139. Hi dear blogger,
    I was really excited to read this blog post. I found it interesting enough to read. A valid link for best SEO courses in India is this, Best SEO Courses in India

    ReplyDelete
  140. It is a very helpful and informative blog post.
    searching for https://tamizhakarthic.com/youtube-mastery-course/
    Youtube Mastery Course

    ReplyDelete
  141. This blog perfectly describes the idea of "model uncertainty. I appreciate the time and effort put into this article. keep sharing helpful information Best GST Courses in India

    ReplyDelete
  142. Hello blogger,
    I just mean to thank you for this wonderful work you have done here. In fact, I am really please to find this blog post on this platform. I enjoyed reading it. Best Content Writing Courses in India

    ReplyDelete
  143. The fact that this article is well described with diagrams and notes on modern uncertainty was much useful for me. Also, if anyone is interested in learning more about Best GST Courses in India, then I would like to recommend you with this article on the Best GST Courses in India – A Detailed Exposition With Live Training. Best GST Courses in India

    ReplyDelete
  144. You have done a great job in discussing the issue of model uncertainty. You have presented different scenarios to highlight how the decisions made by data scientists are impacted by the model uncertainty. The examples you have used are very relevant and easy to understand. It is commendable that you have discussed multiple solutions to address this issue. Overall, you have presented an informative blog which is thought-provoking and insightful. Thank you for sharing your ideas and knowledge. Best Technical Writing Courses in India

    ReplyDelete
  145. You have written an excellent blog on model uncertainty. The topic of model uncertainty is difficult to explain in a simple way, but you have been successful in doing so. Your examples of measuring uncertainty in the form of confidence intervals and hypothesis testing are very informative and interesting. You have also provided practical advice on how to deal with uncertainty in the form of testing different models and picking the most appropriate one. Your insight into this subject is much appreciated. Thank you for making this blog so informative and enjoyable to read. FMVA

    ReplyDelete
  146. Hi very informative and well written. Got to know a lot. You have explained it precisely. Thanks a lot for sharing this information.
    Data Analytics Courses in Kenya

    ReplyDelete
  147. Great job on this blog, you! It was really interesting to read about model uncertainty and the impact it can have on our results. You described the issue in a way that was easy to understand, and you provided a great example to illustrate the importance of understanding this concept. I'm sure this will be a huge help to many people who are just starting out in data analysis. Thanks for your hard work. Article Writing

    ReplyDelete
  148. Thank you for sharing such a nice and interesting blog with us.
    Best Tally Courses in India

    ReplyDelete
  149. Thank you for writing this blog. It was really interesting to read about how uncertainty affects our models. You did a great job of covering the different types of uncertainties that can arise when dealing with models and how to tackle them. I particularly liked the idea of using different methods to quantify uncertainty and the importance of assessing the uncertainty of models. I appreciate your in-depth analysis of the topic and the insight you provide. Thank you for your hard work. Best Technical Writing Courses in India

    ReplyDelete
  150. Your blog post on "A Question of Model Uncertainty" is really well written. You have presented the concept of model uncertainty in a very clear and concise manner. You have discussed the importance of having a good understanding of the data in order to produce a reliable model. Your discussion about the tools used to measure the uncertainty of a model is quite impressive. You have discussed the importance of using these tools and how they can help to improve the model. Great job. Digital Marketing Courses in Glassglow

    ReplyDelete
  151. Hey, I loved your style of writing. please continue up the good work. bookmarking this post to read in future. it was really informational, Detailed and brief.I wil look forward to more articles from you. Data Analytics Courses In Jamshedpur

    ReplyDelete