This blog is a companion to my recent book, Exploring Data in Engineering, the Sciences, and Medicine, published by Oxford University Press. The blog expands on topics discussed in the book, and the content is heavily example-based, making extensive use of the open-source statistical software package R.

Sunday, March 9, 2014

A question of model uncertainty

It has been several months since my last post on classification tree models, because two things have been consuming all of my spare time.  The first is that I taught a night class for the University of Connecticut’s Graduate School of Business, introducing R to students with little or no prior exposure to either R or programming.  My hope is that the students learned something useful – I can say with certainty that I did – but preparing for the class and teaching it took a lot of time.  The other activity, that has taken essentially all of my time since the class ended, is the completion of a book on nonlinear digital filtering using Python, joint work with my colleague Moncef Gabbouj of the Tampere University of Technology in Tampere, Finland.  I will have more to say about both of these activities in the future, but for now I wanted to respond to a question raised about my last post.

Specifically, Professor Frank Harrell, the developer of the extremely useful Hmisc package, asked the following:

            How did you take into account model uncertainty?  The uncertainty resulting from data mining to find nodes and thresholds for continuous predictors has a massive impact on confidence intervals for estimates from recursive partitioning.

The short answer is that model uncertainty was not accounted for in the results I presented last time, primarily because – as Professor Harrell’s comments indicate – this is a complicated issue for tree-based models.  The primary objective of this post and the next few is to discuss this issue.

So first, what exactly is model uncertainty?  Any time we fit an empirical model to data, the results we obtain inherit some of the uncertainty present in the data.  For the specific example of linear regression models, the magnitude of this uncertainty is partially characterized by the standard errors included in the results returned by R’s summary() function.  This magnitude depends on both the uncertainty inherent in the data and the algorithm we use to fit the model.  Sometimes – and classification tree models are a case in point – this uncertainty is not restricted to variations in the values of a fixed set of parameters, but it can manifest itself in substantial structural variations.  That is, if we fit classification tree models to two similar but not identical datasets, the results may differ in the number of terminal nodes, the depths of these terminal nodes, the variables that determine the path to each one, and the values of these variables that determine the split at each intermediate node.  This is the issue Professor Harrell raised in his comments, and the primary point of this post is to present some simple examples to illustrate its nature and severity.

In addition, this post has two other objectives.  The first is to make amends for a very bad practice demonstrated in my last two posts.  Specifically, the classification tree models described there were fit to a relatively large dataset and then evaluated with respect to that same dataset.  This is bad practice because it can lead to overfitting, a problem that I will discuss in detail in my next post.  (For a simple example that illustrates this problem, see the discussion in Section 1.5.3 of Exploring Data in Engineering, the Sciences, and Medicine.)  In the machine learning community, this issue is typically addressed by splitting the original dataset randomly into three parts: a training subset (Tr) used for model-fitting, a validation subset (V) used for intermediate modeling decisions (e.g., which variables to include in the model), and a test subset (Te) used for final model evaluation.  This approach is described in Section 7.2 of The Elements of Statistical Learning by Hastie, Tibshirani, and Friedman, who suggest 50% training, 25% validation, and 25% test as a typical choice.

The other point of this post is to say something about the different roles of model uncertainty and data uncertainty in the practice of predictive modeling.  I will say a little more at the end, but whether we are considering business applications like predicting customer behavior or industrial process control applications to predict the influence of changes in control valve settings, the basic predictive modeling process consists of three steps: build a prediction model; fix (i.e., “finalize”) this model; and apply it to generate predictions from data not seen in the model-building process.  In these applications, model uncertainty plays an important role in the model development process, but once we have fixed the model, we have eliminated this uncertainty by fiat.  Uncertainty remains an important issue in these applications, but the source of this uncertainty is in the data from which the model generates its predictions and not in the model itself once we have fixed it.  Conversely, as George Box famously said, “all models are wrong, but some are useful,” and this point is crucial here: if the model uncertainty is great enough, it may be difficult or impossible to select a fixed model that is good enough to be useful in practice.



Returning to the topic of uncertainty in tree-based models, the above plot is a graphical representation of a classification tree model repeated from my previous two posts.  This model was fit using the ctree procedure in the R package party, taking all optional parameters at their default values.  As before, the dataset used to generate this model was the Australian vehicle insurance dataset car.csv, obtained from the website associated with the book Generalized Linear Models for Insurance Data, by Piet de Jong and Gillian Z. Heller.  This model – and all of the others considered in this post – was fit using the same formula as before:

            Fmla = clm ~ veh_value + veh_body + veh_age + gender + area + agecat

Each record in this dataset describes a single-vehicle, single-driver insurance policy, and clm is a binary response variable taking the value 1 if policy filed one or more claims during the observation period and 0 otherwise.  The other variables (on the right side of “~”) represent covariates that are either numeric (veh_value, the value of the vehicle) or categorical (all other variables, representing the vehicle body type, its age, the gender of the driver, the region where the vehicle is driven, and the driver’s age).

As I noted above, this model was fit to the entire dataset, a practice that is to be discouraged since it does not leave independent datasets of similar character for validation and testing.  To address this problem, I randomly partitioned the original dataset into a 50% training subset, a 25% validation subset, and a 25% test subset as suggested by Hastie, Tibshirani and Friedman.  The plot shown below represents the ctree model we obtain using exactly the same fitting procedure as before, but applied to the 50% random training subset instead of the complete dataset.  Comparing these plots reveals substantial differences in the overall structure of the trees we obtain, strictly as a function of the data used to fit the models.  In particular, while the original model has seven terminal nodes (i.e., the tree assigns every record to one of seven “leaves”), the model obtained from the training data subset has only four.  Also, note that the branches in the original tree model are determined by the three variables agecat, veh_body, and veh_value, while the branches in the model built from the training subset are determined by the two variables agecat and veh_value only.



These differences illustrate the point noted above about the strong dependence of classification tree model structure on the data used in model-building.  One could object that since the two datasets used here differ by a factor of two in size, the comparison isn’t exactly “apples-to-apples.”  To see that this is not really the issue, consider the following two cases, based on the idea of bootstrap resampling.  I won’t attempt a detailed discussion of the bootstrap approach here, but the basic idea is to assess the effects of data variability on a computational procedure by applying that procedure to multiple datasets, each obtained by sampling with replacement from a single source dataset.  (For a comprehensive discussion of the bootstrap and some of its many applications, refer to the book Bootstrap Methods and their Application by A.C. Davison and D.V. Hinkley.)  The essential motivation is that these datasets – called bootstrap resamples – all have the same essential statistical character as the original dataset.  Thus, by comparing the results obtained from different bootstrap resamples, we can assess the variability in results for which exact statistical characterizations are either unknown or impractical to compute.  Here, I use this idea to obtain datasets that should address the “apples-to-apples” concern raised above.  More specifically, I start with the training data subset used to generate the model described in the previous figure, and I use R’s built-in sample() function to sample the rows of this dataframe with replacement.  For an arbitrary dataframe DF, the code to do this is simple:

> set.seed(iseed) 
> BootstrapIndex = sample(seq(1,nrow(DF),1),size=nrow(DF),replace=TRUE 
> ResampleFrame = DF[BootstrapIndex,]

The only variable in this procedure is the seed for the random sampling function sample(), which I have denoted as iseed.  The extremely complicated figure below shows the ctree model obtained using the bootstrap resample generated from the training subset with iseed = 5.





Comparing this model with the previous one – both built from datasets of the same size, with the same general data characteristics – we see that the differences are even more dramatic than those between the original model (built from the complete dataset) and the second one (built from the training subset).  Specifically, while the training subset model has four terminal nodes, determined by two variables, the bootstrap subsample model uses all six of the variables included in the model formula, yielding a tree with 16 terminal nodes.  But wait – sampling with replacement generates a significant number of duplicated records (for large datasets, each bootstrap resample contains approximately 63.2% of the original data values, meaning that the other 36.8% of the resample values must be duplicates).  Could this be the reason the results are so different?  The following example shows that this is not the issue.



This plot shows the ctree model obtained from another bootstrap resample of the training data subset, obtained by specifying iseed = 6 instead of iseed = 5.  This second bootstrap resample tree is much simpler, with only 7 terminal nodes instead of 16, and the branches of the tree are based on only four of the prediction variables instead of all six (specifically, neither gender nor veh_body appear in this model).  While I don’t include all of the corresponding plots, I have also constructed and compared the ctree models obtained from the bootstrap resamples generated for all iseed values between 1 and 8, giving final models involving between four and six variables, with between 7 and 16 terminal nodes.  In all cases, the datasets used in building these models were exactly the same size and had the same statistical character.  The key point is that, as Professor Harrell noted in his comments, the structural variability of these classification tree models across similar datasets is substantial.  In fact, this variability of individual tree-based models was one of the key motivations for developing the random forest method, which achieves substantially reduced model uncertainty by averaging over many randomly generated trees.  Unfortunately, the price we pay for this improved model stability is a complete loss of interpretibility.  That is, looking at any one of the plots shown here, we can construct a simple description (e.g., node 12 in the above figure represents older drivers – agecat > 4 – with less expensive cars, and it has the lowest risk of any of the groups identified there).  While we may obtain less variable predictions by averaging over a large number of these trees, such simple intuitive explanations of the resulting model are no longer possible.

I noted earlier that predictive modeling applications typically involve a three-step strategy: fit the model, fix the model, and apply the model.  I also argued that once we fix the model, we have eliminated model uncertainty when we apply it to new data.  Unfortunately, if the inherent model uncertainty is large, as in the examples presented here, this greatly complicates the “fix the model” step.  That is, if small variations in our training data subset can cause large changes in the structure of our prediction model, it is likely that very different models will exhibit similar performance when applied to our validation data subset.  How, then, do we choose?  I will examine this issue further in my next post when I discuss overfitting and the training/validation/test split in more detail. 


180 comments:

  1. I am trying to use CART algorithms as against conventional statistics, DOE and hypothesis testing. I have got some results and advantages also for problem solving rather than using the data as training set for future prediction.

    Is my understanding correct that uncertainty is more important in cases where we use this training data rather than problem solving approach?

    ReplyDelete

  2. It looks like you spend a lot of effort and time on your blog. I have bookmarked it and I am looking forward to reading new articles. Keep up the good work
    Android Training in Chennai

    ReplyDelete
  3. I am expecting more interesting topics from you. And this was nice content and definitely it will be useful for many people.

    Hadoop Training in Chennai

    ReplyDelete
  4. Wow it is really wonderful and awesome thus it is very much useful for me to understand many concepts and helped me a lot.thus these tips are really awesome and you had a wonderful products.


    Online Reputation Management

    ReplyDelete

  5. What an awesome post, I just read it from start to end. Learned something new after a long time.


    SAP SD training in Chennai

    ReplyDelete
  6. Your thinking toward the respective issue is awesome also the idea behind the blog is very interesting which would bring a new evolution in respective field. Thanks for sharing.

    SEO Company in India

    ReplyDelete
  7. great tips thanks for your impressive enhancement of getting unique style of design a website. and it much more wonderful information to me. keep share many different ideas.
    IELTS coaching in chennai

    ReplyDelete
  8. Truly a very good article on how to handle the future technology. After reading your post,thanks for taking the time to discuss this, I feel happy about and I love learning more about this topic
    Tooth Braces In Chennai

    ReplyDelete
  9. Thank you for sharing such a nice and interesting blog with us. I have seen that all will say the same thing repeatedly. But in your blog, I had a chance to get some useful and unique information. I would like to suggest your blog in my dude circle.
    Jobs in Chennai
    Jobs in Bangalore
    Jobs in Delhi
    Jobs in Hyderabad
    Jobs in Kolkata
    Jobs in Mumbai
    Jobs in Noida
    Jobs in Pune

    ReplyDelete
  10. Thanks for sharing this information and keep updating us. This content is quite informatics to me.
    Hadoop Training in Chennai | Hadoop Training Chennai | Big Data Training in Chennai

    ReplyDelete
  11. Wow.. Thanks much for sharing.. My friend also recommended you so that i can have a helping hand to make my blog as effective as possible.
    Architectural Firms in Chennai
    Architects in Chennai

    ReplyDelete
  12. This content is so informatics and it was motivating all the programmers and beginners to switch over the career into the Big Data Technology. This article is so impressed and keeps updating us regularly.
    Hadoop Training in Chennai | Hadoop Training Chennai | Big Data Training in Chennai

    ReplyDelete
  13. Wonderful blog.. Thanks for sharing informative Post. Its very useful to me.

    Installment loans
    Payday loans
    Title loans

    ReplyDelete
  14. Really Good blog post.provided a helpful information.keep updating...
    Digital marketing company in Chennai

    ReplyDelete
  15. Excellent goods from you, man. I’ve understand your stuff previous to and you’re just too excellent. I actually like what you’ve acquired here, certainly like what you are stating and the way in which you say it. You make it enjoyable and you still take care of to keep it sensible. I can not wait to read far more from you. This is actually a tremendous site..
    Storage space rental singapore
    Cheapest storage space singapore
    Van delivery service singapore

    ReplyDelete

  16. Its a wonderful post and very helpful, thanks for all this information. You are including better information regarding this topic in an effective way.Thank you so much

    Personal Installment Loans
    Payday Cash Advance loan
    Title Car loan
    Cash Advance Loan

    ReplyDelete
  17. wonderful post and very helpful, thanks for all this information. You are including better information regarding this topic in an effective way.Thank you so much
    Web Design Company in Chennai


    ReplyDelete
  18. i really like this blog.And i got more information's from this blog.thanks for sharing!!!!
    SEO Company in Chennai

    ReplyDelete
  19. very informative blog and very useful for readers. making us to learn more from your blog.keep on updating your blog and thanks for sharing
    java training in chennai
    dotnet training in chennai
    cloud -computing training in chennai
    digital marketing training in chennai

    ReplyDelete
  20. Thank you for sharing such a nice and interesting blog with us. But in your blog, I had a chance to get some useful and unique information. I would like to suggest your blog.
    Web development Company in Chennai

    ReplyDelete
  21. this is very nice blog this studying course information very useful to everyone who have learning this information.

    Base SAS Training in chennai

    ReplyDelete
  22. This is a nice article here with some useful tips for those who are not used-to comment that frequently.Thanks for this helpful information I agree with all points you have given to us.I will follow all of them.

    Self Employment Tax
    Tax Preparation Services
    Tax Accountant
    Tax Consultant
    Tax Advisor
    Online Tax services
    Tax Professional
    Online Tax Preparation"

    ReplyDelete
  23. Thank you your blog.it is very useful for me .keep more updates.waiting for your new blog.
    Business Tax Return
    Cpa Tax Accountant
    Tax Return Services

    ReplyDelete
  24. This information is impressive; I am inspired with your post writing style & how continuously you describe this topic.

    Eczema Treatment
    Psoriasis Oil
    Hyperpigmentation Treatment
    Herbal Tonic

    ReplyDelete
  25. It's Really nice post.Thank you for sharing such a blog....!!
    Spoken english Training in Bangalore

    ReplyDelete
  26. Just found your post by searching on the Google, I am Impressed and Learned Lot of new thing from your post. I am new to blogging and always try to learn new skill as I believe that blogging is the full time job for learning new things day by day.
    Eczema Treatment
    Psoriasis Oil
    Hyperpigmentation Treatment
    Herbal Tonic

    ReplyDelete
  27. These ways are very simple and very much useful, as a beginner level these helped me a lot thanks fore sharing these kinds of useful and knowledgeable information.

    Hadoop Training in Chennai

    ReplyDelete
  28. useful and knowledgeable information.
    thank u sharing!!


    This blog was very useful for me waiting for more blog.


    SAP FICO Training in Chennai

    ReplyDelete
  29. A great content and very much useful to the visitors. Looking for more updates in future.

    Selenium Training in Chennai

    ReplyDelete
  30. Its really an Excellent post. I just stumbled upon your blog and wanted to say that I have really enjoyed reading your blog. posts. I hope you post again soon.

    Selenium Training in Chennai

    ReplyDelete
  31. Omrdeals.in is an online platform that allows people to find offers and the greatest deals in and around OMR. It is the directory of the business in and around omr. It helps internet users to surf the required deals and offers.

    Offers in Chennai

    ReplyDelete
  32. It is really very excellent. Because of all given information was wonderful and it's very helpful for me. Thanks for sharing
    SAP Training in Chennai
    SAP ABAP Training in Chennai
    SAP FICO Training in Chennai
    SAP MM Training in Chennai

    ReplyDelete
  33. Thank you for taking the time to provide us with your valuable information. We strive to provide our candidates with excellent care.As always, we appreciate your confidence and trust in us.
    Software Testing Training in Chennai
    SEO Training in Chennai
    Informatica Training in Chennai
    Digital Marketing Training in Chennai

    ReplyDelete
  34. Thanks for sharing, Really it is an amazing article I had ever read. I hope it will help a lot for all. Thank you so much for this amazing posts and please keep update like this.

    OFFERS IN CHENNAI

    ReplyDelete
  35. These ways are very simple and very much useful, as a beginner level these helped me a lot thanks fore sharing these kinds of useful and knowledgeable information.
    Also Check out the : https://www.credosystemz.com/training-in-chennai/best-data-science-training-in-chennai/

    ReplyDelete
  36. Usually I do not read post on blogs, but I would like to say that this write-up very forced me to try and do it! Your writing style has been surprised me. Great work admin..Keep update more blog..
    Economics Journal
    Journal Of Ecology
    Thesis by publication
    National Journal
    Journal Of Information Technology

    ReplyDelete
  37. Benefits of reading your post. This will help in the maintenance of our health.The daily routine will help you lose weight quickly and safely.My life is totally transformed after I followed this diet.I feeling great about myself.

    Herbalife in Chennai
    Herbalife Nutrition Products
    Nutritionclub in Chennai
    Weight Loss in Chennai
    Weight Gain in Chennai

    ReplyDelete

  38. I have really happy to these reading your post. This product control and maintenance of our health.The daily routine can assist you weight lose quickly and safely.My life is completely reworked once I followed this diet.I feeling nice concerning myself.

    Herbalife in Chennai
    Herbalife Nutrition Products
    Nutrition centers in Chennai
    Weight Loss in Chennai
    Weight Gain in Chennai

    ReplyDelete
  39. Your site is amazing and your blogs are informative and knlowledgeble to my websites.This is one of the best tips in my life.I have in quite some time.Nicely written and great info.Thanks to share the more informations.

    Seo Experts
    Seo Company
    Web Designing Company
    Digital Marketing
    Web Development Company
    Apps Development

    ReplyDelete
  40. SQream Technologies provides you with a state of the art software which combines modern GPU technology (Graphic Processing Units) with the best practices in today’s Big Data platforms, providing up to 100x faster insights from data.
    Bigdata Training in Chennai OMR

    ReplyDelete
  41. SQream Technologies provides you with a state of the art software which combines modern GPU technology (Graphic Processing Units) with the best practices in today’s Big Data platforms, providing up to 100x faster insights from data.
    Bigdata Training in Chennai OMR

    ReplyDelete
  42. Best R Programming Training in Bangalore offered by myTectra. India's No.1 R Programming Training Institute. Classroom, Online and Corporate training in R Programming
    r programming training

    ReplyDelete
  43. myTectra Placement Portal is a Web based portal brings Potentials Employers and myTectra Candidates on a common platform for placement assistance

    ReplyDelete
  44. Selenium is one of the most popular automated testing tool used to automate various types of applications. Selenium is a package of several testing tools designed in a way for to support and encourage automation testing of functional aspects of web-based applications and a wide range of browsers and platforms and for the same reason, it is referred to as a Suite.

    Selenium Interview Questions and Answers
    Javascript Interview Questions
    Human Resource (HR) Interview Questions

    ReplyDelete
  45. This is a great post. I like this topic.This site has lots of advantage. I found many interesting things from this site. It helps me in many ways.Thanks for posting this again.
    Best ERP Software
    ERP Software In India
    ERP Companies In Chennai
    ERP Software In Chennai

    ReplyDelete
  46. Amazing article. Your blog helped me to improve myself in many ways thanks for sharing this kind of wonderful informative blogs in live. I have bookmarked more article from this website. Such a nice blog you are providing ! Kindly Visit Us

    R Programming institutes in Chennai | R Programming Training in Chennai | R Programming Course Fees | R Language training in Chennai

    ReplyDelete
  47. Great post! I am actually getting ready to across this information, It's very helpful for this blog.Also great with all of the valuable information you have Keep up the good work you are doing well. big data training in Velachery | Hadoop Training in Chennai | big data Hadoop training and certification in Chennai | Big data course fees |

    ReplyDelete

  48. https://tvskybox.com
    7500 iptv channels only 5 euro/usd
    /europe/usa/asia/arabic/latino/russian channels....

    ReplyDelete
  49. Very useful and informative content has been shared out here, Thanks for sharing it.
    Visit Learn Digital Academy for more information on Digital marketing course in Bangalore https://www.learndigital.co/.

    ReplyDelete
  50. Laminated Doors manufacturer in hubli
    Thanks for giving great kind of information. So useful and practical for me. Thanks for your excellent blog, nice work keep it up thanks for sharing the knowledge.

    ReplyDelete
  51. led lawn lights in delhi
    Thanks for giving great kind of information. So useful and practical for me. Thanks for your excellent blog, nice work keep it up thanks for sharing the knowledge.

    ReplyDelete
  52. hello sir,
    thanks for giving that type of information. Really enjoyed this blog post. Really looking forward to reading more. Much obliged.
    digital marketing company in delhi

    ReplyDelete
  53. Thanks for giving great kind of information. So useful and practical for me. Thanks for your excellent blog, nice work keep it up thanks for sharing the knowledge.
    Modular Kitchen in Noida

    ReplyDelete
  54. Thanks for giving great kind of information. So useful and practical for me. Thanks for your excellent blog, nice work keep it up thanks for sharing the knowledge.
    Modular Kitchen in Noida

    ReplyDelete
  55. nice work keep it up thanks for sharing the knowledge.Thanks for sharing this type of information, it is so useful.
    Epoxy Grout manufacturer

    ReplyDelete

  56. The post was really very good.Thanks for sharing
    prestige elysian

    ReplyDelete
  57. I feel happy about and learning more about this topic. keep sharing your information regularly for my future reference. This content creates a new hope and inspiration with in me. Thanks for sharing article like this. the information which you have provided is better then other blog.
    Best IELTS Coaching in Dwarka

    ReplyDelete
  58. Nice blog..! I really loved reading through this article. Thanks for sharing such a amazing post with us and keep blogging... Also Checkout: best angularjs training institute in chennai | angularjs training in omr | angularjs training in chennai | angularjs training in velachery |

    ReplyDelete
  59. I am really happy to say this I am deeply read your article, I am searching like this type valuable information, it’s really helpful for me, I am happy to found it, thank you so much for share this blog, great work, keep sharing like this type of article, thank you so much for read my comment, if any one searching website designing company in India please visit my website
    Web Designing company

    ReplyDelete
  60. This is the exact information I am been searching for, Thanks for sharing the required infos with the clear update and required points. To appreciate this I like to share some useful information regarding Microsoft Azure which is latest and newest,

    Regards,
    Ramya

    Azure Training in Chennai
    Azure Training Center in Chennai
    Best Azure Training in Chennai
    Azure Devops Training in Chenna
    Azure Training Institute in Chennai
    Azure Training in Chennai OMR
    Azure Training in Chennai Velachery
    Azure Online Training
    Azure Training in Chennai Credo Systemz
    DevOps Training in Chennai Credo Systemz

    ReplyDelete
  61. I feel happy about and learning more about this topic. keep sharing your information regularly for my future reference. This content creates new hope and inspiration within me. Thanks for sharing an article like this. the information which you have provided is better than another blog.
    IELTS Coaching in Dwarka mor

    ReplyDelete
  62. Great Post,really it was very helpful for us.
    Thanks a lot for sharing!
    I found this blog to be very useful!!
    Hadoop training in Bangalore

    ReplyDelete
  63. I really loved reading your blog. I also found your posts very interesting. In fact after reading, I had to go show it to my friend and he enjoyed it as well!
    Authorized apple service center in Chennai | apple service center in chennai | Authorized apple service center in Chennai | apple service center in chennai | ipad service center in chennai | iphone water damage service in chennai

    ReplyDelete
  64. Nice post. Thanks for sharing! I want people to know just how good this information is in your article. It’s interesting content and Great work.
    Thanks & Regards,
    VRIT Professionals,
    No.1 Leading Web Designing Training Institute In Chennai.

    And also those who are looking for
    Web Designing Training Institute in Chennai
    SEO Training Institute in Chennai
    Photoshop Training Institute in Chennai
    PHP & Mysql Training Institute in Chennai
    Android Training Institute in Chennai

    ReplyDelete
  65. Great efforts put it to find the list of articles which is very useful to know, Definitely will share the
    same to other forums.
    big data hadoop training cost in chennai | hadoop training in Chennai | best bigdata hadoop training in chennai | best hadoop certification in Chennai

    ReplyDelete
  66. Vanskeligheter( van bi ) vil passere. På samme måte som( van điện từ ) regnet utenfor( van giảm áp ) vinduet, hvor nostalgisk( van bướm ) er det som til slutt( van cửa ) vil fjerne( van công nghiệp ) himmelen.

    ReplyDelete
  67. Si el agua cae al lago, desaparecerá( phụ kiện tủ bếp ). Pero si cae a la hoja de( phụ kiện tủ áo ) loto, brillará como una joya. Caer igual pero( thùng gạo thông minh ) estar con alguien es importante.

    ReplyDelete
  68. agar aapka husband aapse ladai karta hai or aap karna nhi chatey toh aap jarur apne pati ko apne vash main kar sakty ho पति को वश में करने के टोटके kare

    ReplyDelete
  69. Good blog article
    Sanjary Kids is one of the best play school and preschool in Hyderabad,India. Give your child the best preschool experience by choosing the best playschool of Hyderabad in Abids. we provide programs like Play group,Nursery,Junior KG,Senior KG,and provides Teacher Training Program.
    Preschool in hyderabad

    ReplyDelete
  70. Usually I never comment on blogs but your article is so convincing that I never stop myself to say something about it. You’re doing a great job Man,Keep it up.
    study in canada

    ReplyDelete
  71. I really enjoyed your blog Thanks for sharing such an informative post.Looking For Some More Stuff.

    shuttering works

    ReplyDelete
  72. I really enjoyed your blog Thanks for sharing such an informative post.Looking For Some More Stuff.

    best seo company in bangalore SSS digital Marketing

    ReplyDelete
  73. Good blog posts information by the author

    Sanjary Academy is the best Piping Design institute in Hyderabad, Telangana. It is the best Piping design Course in India and we have offer professional Engineering Courses like Piping design Course, QA/QC Course, document controller course, Pressure Vessel Design Course, Welding Inspector Course, Quality Management Course and Safety Officer Course.
    Piping Design Course
    Piping Design Course in Hyderabad ­
    Piping Design Course in India­

    ReplyDelete
  74. Usually I never comment on blogs but your article is so convincing that I never stop myself to say something about it. You’re doing a great job Man,Keep it up.
    Study Abroad in New Zealand

    ReplyDelete
  75. Good blog information I liked it

    Sanjary Academy is the best Piping Design institute in Hyderabad, Telangana. It is the best Piping design Course in India and we have offer professional Engineering Courses like Piping design Course, QA/QC Course, document controller course, Pressure Vessel Design Course, Welding Inspector Course, Quality Management Course and Safety Officer Course.
    Piping Design Course
    Piping Design Course in Hyderabad ­
    Piping Design Course in India­

    ReplyDelete
  76. Thanks for provide great informatic and looking beautiful blog, really nice required information & the things i never imagined and i would request, wright more blog and blog post like that for us. Thanks you once agianMarriage certificate in delhi
    Marriage certificate in ghaziabad
    Marriage registration in gurgaon
    Marriage registration in noida
    special marriage act
    Marriage certificate online
    Marriage certificate in mumbai
    Marriage certificate in faridabad
    Marriage certificate in bangalore
    Marriage certificate in hyderabad thanks once again to all.

    ReplyDelete
  77. Great blog information and understandable

    Sanjary Academy provide pressure vessel design,quality management system course, piping design course, qa/qc course and document controller course.
    Welding Inspector Course
    Safety officer course
    Quality Management Course
    Quality Management Course in India

    ReplyDelete
  78. Nice post. It is really interesting. Thanks for sharing the post!
    Buy AC Online | Smart LED TV
    Laptops for Sale | Best Inverter AC

    ReplyDelete
  79. Webclick Digital Pvt. Ltd. is one of the Website Designing Company In India that successfully made themselves a brand in the fierce competition. Our aim and focus is on creating beautiful web designs that speak for your business and justify your actions to the audience. Start a discussion with our experts to know our services better.

    ReplyDelete
  80. Nice Information
    Yaaron Studios is one of the rapidly growing editing studios in Hyderabad. We are the best Video Editing services in Hyderabad. We provides best graphic works like logo reveals, corporate presentation Etc. And also we gives the best Outdoor/Indoor shoots and Ad Making services.
    Best video editing services in Hyderabad,ameerpet
    Best Graphic Designing services in Hyderabad,ameerpet­
    Best Ad Making services in Hyderabad,ameerpet­

    ReplyDelete
  81. Thanks for sharing information
    "Sanjary Academy provides excellent training for Piping design course. Best Piping Design Training Institute in Hyderabad,
    Telangana. We have offer professional Engineering Course like Piping Design Course,QA / QC Course,document Controller
    course,pressure Vessel Design Course, Welding Inspector Course, Quality Management Course, #Safety officer course."
    Piping Design Course in India­
    Piping Design Course in Hyderabad
    Piping Design Course in Hyderabad
    QA / QC Course
    QA / QC Course in india
    QA / QC Course in Hyderabad
    Document Controller course
    Pressure Vessel Design Course
    Welding Inspector Course
    Quality Management Course
    Quality Management Course in india
    Safety officer course

    ReplyDelete



  82. Soma pill is very effective as a painkiller that helps us to get effective relief from pain. This cannot cure pain. Yet when it is taken with proper rest, it can offer you effective relief from pain.
    This painkiller can offer you relief from any kind of pain. But Soma 350 mg is best in treating acute pain. Acute pain is a type of short-term pain which is sharp in nature. Buy Soma 350 mg online to get relief from your acute pain.

    https://globalonlinepills.com/product/soma-350-mg/


    Buy Soma 350 mg
    Soma Pill
    Buy Soma 350 mg online



    Buy Soma 350 mg online
    Soma Pill
    Buy Soma 350 mg

    ReplyDelete
  83. Nice blog..! I really loved reading through this article. Thanks for sharing.You done a great job.Oracle Applications Training in Bangalore

    ReplyDelete
  84. Thank you for excellent article.You made an article that is interesting.
    AWS Solutions Architect courses in Bangalore with certifications.
    https://onlineidealab.com/aws-training-in-bangalore/


    ReplyDelete
  85. Thank you for excellent article.You made an article that is interesting.
    Digital marketing course in Bangalore with Live Projects. 100% placement, 20+ modules, 10+ certifications, Great discounts on course fees.
    https://onlineidealab.com/digital-marketing-courses-in-bangalore/

    ReplyDelete
  86. The first thing I look for is personalization. This is so easy, I really enjoyed this post. Check out this unrelated link. I wrote about the IELTS.
    IELTS Coaching in Dwarka
    IELTS Coaching in Dwarka

    ReplyDelete
  87. keep up the good work. this is an Assam post. this to helpful, i have reading here all post. i am impressed. thank you. this is our digital marketing training center. This is an online certificate course
    digital marketing training in bangalore | https://www.excelr.com/digital-marketing-training-in-bangalore

    ReplyDelete
  88. Digital Yogi - Get Stunning Website Design, Web Development, SEO, E-commerce Development, Content Writer, Redesign Website, responsive Design, Wordpress Development, Digital Marketing Services. Email us - hello@digitalyogi.co.in and Call: 9999156257
    E-Commerce website design,Best website design company,SEO,Internet (https://www.digitalyogi.co.in/)

    ReplyDelete
  89. Thanks for one marvelous posting! I enjoyed reading it; you are a great author. I will make sure to bookmark your blog and may come back someday. I want to encourage that you continue your great posts.






    learn digital marketing

    ReplyDelete
  90. Taldeen is one of the best plastic manufacturing company in Saudi Arabia. They are manufacturing Handling Solutions Plastic products like Plastic Pallets and plastic crates. Here is the link of the product
    Handling Solutions
    Plastic Pallets

    ReplyDelete
  91. Here is the details of best BSc Medical Imaging Technology Colleges in Bangalore. You can get the college details from the below link. BSc Medical Imaging Technology Course is one of the best demanding course in recent times in India
    BSc Medical Imaging Technology Colleges In Bangalore

    ReplyDelete
  92. Christian College Bangalore providing BSc Medical Imaging Technology Course. Here is the link about the details of BSc Medical Imaging Technology. You can click the below link for more information about BSc Medical Imaging Technology.
    BSc Cardiac Care Technology Colleges In Bangalore

    ReplyDelete
  93. Christian College Bangalore providing BSc Optometry Course. Here is the link about the details of BSc Optometry. You can click the below link for more information about BSc Optometry. BSc Optometry is one of the most demanding course in recent times.
    Optometry Colleges In Bangalore

    ReplyDelete
  94. BBA Aviation course is the best (Most Demanded) management course in India. Here, Christian College Bangalore providing BBA Aviation course. You can get the details of Christian College BBA Aviation from the below mentioned link. If you are interested in BBA Aviation, just visit the below link to know about BBA Aviation.
    BBA Aviation Colleges In Bangalore

    ReplyDelete
  95. GrueBleen is one of the Branding and Marketing agency Based in Riyadh- Saudi Arabia. The main functions of GrueBleen is Advertising, Branding, Marketing, Office Branding, Exhibition Management and Digital Marketing. Visit the below link to know more about GrueBleen Creative Club.
    Branding Agency Riyadh
    Marketing Agency Riyadh

    ReplyDelete
  96. Agriculture Solutions – Taldeen is a plastic manufacturing company in Saudi Arabia. They are manufacturing agricultural plastic products like greenhouse cover and hay cover. Visit the below link to know more details
    Agriculture Solutions
    Greenhouse Cover

    ReplyDelete
  97. Medical Imaging Technology – One of the most demanding allied health science course in recent times in India. Check out the details of Best BSc Medical Imaging Technology Colleges Details with the following link.
    BSc Medical Imaging Technology Colleges In Bangalore

    ReplyDelete
  98. BSc Perfusion Technology – If you are looking to study BSc Perfusion Technology in Bangalore, just check out the following link. In that link you can get the details of Best BSc Medical Imaging Technology colleges in Bangalore
    BSc Perfusion Technology Colleges in Bangalore

    ReplyDelete
  99. GrueBleen – One of the best social media marketing agency in Riyadh- Saudi Arabia. Visit here for the all service details of GrueBleen.
    Social Media Marketing Agency

    ReplyDelete
  100. Awesome article, it was exceptionally helpful! I simply began in this and I'm becoming more acquainted with it better! Cheers, keep doing awesome
    All University BCOM 1st Year TimeTable 2020

    ReplyDelete
  101. This was definitely one of my favorite blogs. Every post published did impress me. ExcelR Digital Marketing Class In Pune

    ReplyDelete