When building a website, regardless of the technologies used, sometimes you want certain pages to be publicly accessible, but not show up in results from in-site searches for one reason or another. The problem with execution of this with a community driven content management system such as Drupal is that you are working with modules written by different developers and every one does things a little differently … even within the guidelines.
When I first heard that Drupal was going to be a fundamental change in the entire architecture by being based on the Symfony framework for version 8, I was a bit concerned that I would not be able to get up to speed on the new features quickly. I actually sat on the sidelines about remaining in Drupal development during the alpha and most of the beta phases. I dabbled in some of the releases but didn't know enough about Symfony and the articles I found online were usually written about some now obsolete API as the CMS went through many, many, many API changes during the five years it was in development.
The primary purpose of any content management system is to allow administrators to create content without having to have extensive experience with HTML or even an understanding of how a web server works, Drupal is no different. In today's online world, however, the definition of content has expanded far beyond text on the screen. Users want pictures, manuals, documentation, and audio and video content.
But what do you do if you want to restrict access to certain files from general public consumption? How can you hide your files from search engines?
One of the most frustrating things about Drupal 8 is figuring out how to do things "the new way". watchdog() had been my faithful friend all through my development in D7. But, as famous Jedi once said, "You must unlearn what you have learned."
Let's face it, it doesn't matter how pretty your new website is or what kind of bells and whistles you've built to get business done. If your users can't find what they are looking for, they will leave and find it somewhere else! That's why it's critical that your site have some sort of search capabilities.