Tag Archives: Search Engines

Alternatives to Google: Carrot Search and eTools.ch

July 13, 2016 Karen Blakeman 2 Comments

Two of the services I cover in my workshop for researchers on alternatives to Google are Carrot Search and eTools.ch, and recently one of the people who had attended the session in April asked me to confirm what Carrot Search used to provide its main results. Strictly speaking, neither Carrot Search nor eTools are Google free: eTools is a metasearch tool that has Google as one of its sources and Carrot Search uses eTools for its web search. At the start of the year, Carrot Search offered 7 options for searching under tabs across the top of the search screen including Web, “wiki”, Bing, News, Images, PubMed and Jobs. Web search used eTools.ch to provide the results.

The range of options has now been reduced to just three: the more transparently labelled eTools Web Search, PubMed and Jobs.

This makes sense as the number of accesses to Bing via the api was always limited and I could never get the news or images options to work. eTools in any case is a metasearch engine covering 17 tools including Google, Bing and Wikipedia so the extra Carrot Search tabs did seem to be unnecessary. The full list can be seen on the eTools home page.

This is where it gets interesting. It appears that Carrot Search does not just copy the results from a search on eTools. I ran a search on Brexit in Carrot Search and compared the results from eTools Worldwide and eTools United Kingdom. All of the sets were different so Carrot Search must be doing some additional analysis and processing.

Carrot Search doesn’t just list the results but also organises them into topics or Folders that are displayed on the left hand side of the screen. These can be a useful way of narrowing down your search.

Carrot Search offers two other ways of displaying results: Circles and Foam Tree.

Carrot Search Foam Tree – 13th July 2016

Both show the density of terms in the top 100 results and allow you to click on an area to add the term or phrase to the search. In addition I am finding that the Foam Tree is an interesting way of monitoring changes in news coverage and social media discussions on a topic, product or company. Yesterday, when I ran the search on Brexit, there was an area representing Theresa May. Today, that had been replaced with one for David Cameron. I assume that is because the news coverage has been concentrating on David Cameron’s last day as Prime Minister and his last Prime Minister’s Questions (PMQ) in Parliament . Later he goes to see the Queen to officially resign as Prime Minister. Tomorrow, with Theresa May as our new Prime Minister and a new Cabinet, the Foam Tree could have a very different structure so I shall be looking at it periodically to see if and how it reflects changes in events.

As I mentioned earlier eTools.ch, which is behind the main Carrot Search web search, is a metasearch engine covering 17 tools. It also has options to select a country from a drop down list (Worldwide, Swtzerland, Liechtenstein, Germany, Austria, France, Italy, Spain, UK) and a language (All, English, German, French, Italian, Spanish). Either or both of these give you completely different views and opinions on a subject.

eTools - Switzerland, all languages — **eTools – Switzerland, all languages**

eTools_CH_French — **eTools – Switzerland, French**

eTools - Spain, all languages — **eTools – Spain, all languages**

It is a convenient way of gathering a range of foreign language information, especially on European events, and is easier than searching individual country versions of Google or Bing. The disadvantages are that the range of countries and languages is limited and many of the articles will not be in English. Nevertheless, I often find it helpful at the start of a piece of research as I get a general feel for the type and range of information that is available.

Carrot Search and eTools.ch are just two of the tools that I cover in my workshop on alternatives to Google. If you are interested in finding out more, the next session is being organised by UKeiG and will be held in London on Wednesday, 7th September 2016. Further details are available on the UKeiG website.

privacy, Search Engines

“Do not track” does not mean anonymous browsing

December 15, 2015 Karen Blakeman 2 Comments

A question that I’m often asked is “do search engines that don’t track your search history also anonymize your IP address?” DuckDuckGo is the first search tool that often springs to mind with respect to “do not track”. It does not store searches, web history or IP addresses when you use it to search. Also, it does not pass on the search terms you used to the sites that you visit. However, the sites that you visit will still be able to see your IP address. See https://duckduckgo.com/privacy for further details.

Ixquick (http://ixquick.com/) and StartPage (http://startpage.com/) are similar but have an additional feature that gives you the option to display a page from the results list using a proxy. Run the search as normal and you’ll see the usual set of results. Next to each result you should see a “proxy” link. Click on that and you go through a proxy server making you invisible to the website you are visiting.

Any links that you subsequently click on and which are on the same site also go through the proxy. As soon as you follow any links that take you off that site then you are warned that you that you will be “unproxied”.

The disadvantages of using the proxy option are that it can be slower, some functions on the page may not work, and I have come across some pages that do not display at all.

Search Engines

Farewell AltaVista

July 1, 2013 Karen Blakeman

Yahoo is finally pulling the plug on AltaVista on July 8th. It appears as a one line entry in Yahoo’s latest list of closures (http://yahoo.tumblr.com/post/54125001066/keeping-our-focus-on-whats-next) with the comment “Please visit Yahoo! Search for all of your searching needs”. AltaVista was started by Digital Equipment in 1995 and quickly became the default search engine for many of us. I still meet people who have remained loyal to AltaVista even though it lost its unique search features a long time ago. Danny Sullivan has written a short history and eulogy for the search engine at http://searchengineland.com/altavista-eulogy-165366 – “A Eulogy for AltaVista, The Google of its Time”. Great though it was, some of us had already defected to the Inktomi powered search engine HotBot by the time Google had arrived on the scene. Alas, HotBot is now a shadow of its former self and AlltheWeb, which Yahoo had also acquired, was closed down in April 2011.

I’ve unearthed the AltaVista chapter and summary that I wrote for an early edition of Search Strategies. The chapter is at http://www.rba.co.uk/search/altavista/AltaVistaChapter.pdf and the summary sheet at http://www.rba.co.uk/search/altavista/avsumm.pdf

Google

Now you see it, now you don’t

March 27, 2013 Karen Blakeman 4 Comments

April is going to be a very busy month for me this year. As well as speaking at conferences I am also giving six full day workshops so am having to prepare the presentations, handouts and notes well in advance. When it comes to the Google sessions the material the delegates receive never matches exactly what they see on their screens during the practicals. That’s par for the course where Google is concerned and it’s a great way of getting across to people how Google ~~messes up~~ enhances search results. The problem I had yesterday, and am still having this morning, is that Google seems to have dumped me into several major ‘live experiments’ and results keep changing second by second. The consequence is that it is impossible for me to pull together a set of consistent screen shots but I, and the delegates, will just have to live with that. And it makes a good story on the day!

If you don’t know what Google’s ‘live experiments’ are the YouTube video at http://www.youtube.com/watch?v=J5RZOU6vK4Q will enlighten you. In essence, Google tests out changes to its search and ranking algorithms on users before deciding whether or not to go ahead with the changes. It could be me or you who ends up being one of Google’s lab rats. We are not asked if we want to be part of the test nor are we told. Most of the time the changes are so minor that we don’t notice the difference but occasionally they lead to some very bizarre results. See my blog posting from a couple of years ago when Google decided that coots were really lions (http://www.rba.co.uk/wordpress/2011/02/12/google-decides-that-coots-are-really-lions/). What I’ve been seeing over the last couple of days is not in that league but extremely irritating all the same.

One of test searches is fairly straightforward – copper extraction north wales. This is what I saw:

Search in Google Chrome - no emboldened words in the extract — Search in Google Chrome – no emboldened terms in the extract

What’s wrong with that you might ask. At first glance it looks as though Google is dropping terms from my search because none of them are emboldened in the extracts. On closer inspection, though, the terms and their synonyms are present. I ran Verbatim on the search and saw a similar set of results with no emboldening apart from words in the title.

I use Chrome as my default browser and wondering if it was an issue with that I tried Firefox. The emboldened terms reappeared.

Search in Firefox - emboldened terms present in the extracts — Search in Firefox – emboldened terms present in the extracts

Internet Explorer also displayed emboldened terms.

I went back to Chrome and ran the search in an Incognito window. The search terms appeared emboldened in the extracts.

Thinking the problem was due to me being signed in to a Google account I signed out and ran the search. No emboldened terms. I cleared the cache and cookies. No emboldened terms. I disconnected Chrome from my Google account. No emboldened terms. I disabled all of the extensions. No emboldened terms. It was clear that Google was not going to show me emboldened terms when using a normal Chrome window. Why is it so important? Because it is a quick way of initially assessing the relevance of the results. No emboldened terms in the extract suggests that they were not found in the text of the page. If this is indeed an experiment and not a local glitch on my system, and Google decides to roll this out to all users we are all going to waste a lot of time wading through irrelevant results.

On to possible experiment number 2. Google sometimes ignores the setting that tells it how many results to display on a page. I have set mine to 100 but occasionally it reverts to just 10. Refreshing the page or going into settings and saving them again usually works for me. This is a minor irritant, unlike experiment number 3.

Google has started showing just six results for some searches. Phil Bradley is one who is definitely not impressed with this (Google results – down to 6 on a page and most of those are wrong! ). Then I started seeing it for some of my own searches.

I didn’t see 6 results but 4! (As an aside, the emboldened search terms in the extract have returned). The fifth was a result for similar searches with an annotation that indicated ‘& co second hand’ had been omitted. A couple of the results were OK-ish but I was hoping for more detailed information. Is there really so little information for this query? Like Phil, when I clicked on to the next page I was back to sensible results. Unlike Phil, using Verbatim on the search worked for me and overrode the experiment, so again I was back to sensible results. This morning, I could not replicate the 6 results per page display.

Experiment number 4: annotations below the extracts. Some of these annotations look like headings from the pages themselves but others are not. I cannot replicate what I saw yesterday and didn’t take any screenshots of this one. I am definitely sure I didn’t dream it because a couple of my network on Twitter have reported similar experiences.

This continual round of disappearing, reappearing, disappearing “features” is infuriating. Yes, we can all go off and use other search engines but there are times when the type of content and level of coverage tempts us back. You do have to know how to use the advanced search commands to get anything sensible out of Google, but even then success is not guaranteed. This is an area I concentrate on in my workshops. The next one on Google is being organised by UKeiG in Manchester (see the UKeiG web site for details). The title “Make Google behave: techniques for better results” may seem a little overoptimistic given my own and other people’s experiences, but there are plenty of tricks we can employ to get usable results.

Given that Google is now just over 13 years old and a teenager it is not surprising that it has become somewhat truculent. It’s when it starts going through the silent grunting phase that we need to really start worrying.

Google, Search Engines

How search works – sort of

March 4, 2013 Karen Blakeman 2 Comments

Google has put together a site showing how Google search works (http://www.google.com/insidesearch/howsearchworks/thestory/). The main page is a scrolling animated graphic that just gives you some elementary facts but there are links to more detailed information and videos on the main topics of crawling and indexing, the searching and ranking algorithms, fighting spam and Google’s general policies. They are a useful set of pages for anyone who does not already know the basics of how Google works, but if you are looking for something that tells you how to get sensible results from Google you’ll be disappointed. As Phil Bradley says:

“…. boils down to ‘we find some stuff, do magic to it, filter out the crap that our magic didn’t get and then give it to you.’ Yes folks, an entire site to say that. Wasted opportunity.”

Google, Search Engines, Search Strategies, Top 10 Search Tips

Top tips for finding research information

February 27, 2013 Karen Blakeman 3 Comments

This week I was in Canterbury leading a workshop and discussion on Google and Google Scholar for finding research information. Although the emphasis was on Google we also covered other specialist tools designed to search for scientific and research information. We also had an interesting discussion on h-index, other citation indices and services such as ORCID and ResearchGate. The slides for the session are available on authorSTREAM (http://www.authorstream.com/Presentation/karenblakeman-1706478-google-scholar-research-information/), Slideshare (http://www.slideshare.net/KarenBlakeman/scholar-research-information) and temporarily at http://www.rba.co.uk/as/.

Anyone who has attended one of my workshops knows that I ask the group to propose at the end of the session their top tips. These are the Canterbury group’s top 10 tips.

1. What’s going on?
Try and find out what’s going on behind the scenes and how the different search tools work. For example, Google and Google Scholar are quite different in the way they manage your search. Understanding how they operate means that you can adapt your search strategy accordingly and also manage your expectations; for example Google Scholar does not use the publishers’ meta data so author and date search are unreliable.

2. Personalisation and ‘unpersonalisation’
Google personalises your search based on past activity, who is in your social networks,and a whole host of other ‘stuff’. You can quickly ‘unpersonalise’ your results by using a separate browser window that does not use cookies or your web history as part of the search algorithm.

If you use Chrome as your browser, open what is called an incognito window. In the top right hand corner of your screen there is an icon with three lines. Click on it and from the drop down menu select New incognito window. Alternatively press the Ctrl Shift N keys on your keyboard

If you use Firefox, from the menu at the top of the screen select Tools followed by Start Private Browsing.

In Internet Explorer select Tools followed by InPrivate Browsing. If you cannot see InPrivate under Tools try looking under the Safety option.

3. Advanced search commands
Use Google advanced commands such as filetype: to focus on PDFs, presentations, spreadsheets containing data and site: to look for information on just one site or a range of sites such as UK government. Although the advanced search screen has boxes for you to fill in for the commands the file format or filetype option is limited. It does not include options for the newer Microsoft Office formats such as .pptx and xlsx. Use filetype: as part of your search strategy, for example:

nasa dark energy dark matter filetype:pptx

Google Scholar commands are more limited – see slide 28 of the presentation.

4. intext:
Google automatically looks for variations on your terms and sometimes omits words from your search if it thinks the number of results is too low. Prefixing a term with intext: tells Google that it must be included in your search and exactly as you have typed it in. For example:

UK public transport intext:biodiesel statistics

tells Google that biodiesel must be included in the search and exactly as typed in.

5. Reading Level
Use Reading level if Google is failing to return any research oriented documents for a query. Run the search and from the menu above the results select Search tools, All results and then from the drop menu Reading level. Options for switching between basic, intermediate and advanced reading levels should then appear just above the results. Google does not give much away as to how it calculates the reading level and it has nothing to do with the reading age that publishers assign to publications. It seems to involve an analysis of sentence structure, the length of sentences, the length of the document and whether scientific or industry specific terminology appears in the page.

6. Date options
In Google web search, use the date options in the menus at the top of the results page to restrict your results to information that has been published within the last hour, day, week, month, year or your own date range. Click on Search tools, then Any time and select an option. This works best with news, discussion boards, and blogs and web sites that use blogging software to generate pages but Google is getting better at identifying the correct date of a web page.

Google Scholar handles publication dates differently. On the results page you can select a date range from the menu on the left hand of the page. Alternatively, you can run a Google advanced search and enter your publication years. However, Google Scholar looks for publication years in the area of the document where the date is most likely to be. As a result it may identify a page number or part of an author’s address as a year!

7. Google Scholar alerts
To be used with caution as the searches periodically stop without warning, and so have to be set up again, and they sometimes include documents that are several years old. Whatever your search you can set up an alert by selecting Create alert from the menu on the left hand side of the results page.

If the author has created a profile on Google Scholar, from their profile page you can follow new articles and/or new citations for that author. From past experience I warn you that this is not entirely reliable.

8. Metrics – top publications
Although it claims to search all scholarly literature Google Scholar does not always cover all of the key journals in a subject area. There is no complete source list but there is a top publications for subjects and languages under the ‘Metrics’ link in the upper right hand corner of the Scholar home page.

9. Microsoft Academic Search – visualisations
Microsoft Academic Search (http://academic.research.microsoft.com/) is a direct competitor to Google Scholar. The site is sometimes slow to load and it often assigns authors to the wrong institution. Nevertheless, the visualisations such as the co-author and citation maps can be useful in identifying who else is working in a particular area of research. The visualisations can be accessed by clicking on the Citation Graph image to the left of the search results or author profile.

Microsoft academic search citation graph — Author Citation Graph

10. Mednar visual
Deep Web Technologies has developed in conjunction with various institutions a number of science and research specific portals, some of which are publicly available. The sources that they cover are different but they all have similar search and display options. Results are automatically ranked by relevance but this can be changed to date, title or author. In addition to the standard relevance ranked list of results the portals create clusters of topics on the left hand side of the screen. The topics include broad subject headings, authors, publications, publishers, and year of publication and are a useful tool for narrowing down a search. Some of the portals, such as Mednar (http://mednar.com/), offer a clickable ‘visual’ of topics and sub-topics.

Google, Search Engines, Search Strategies

Million Short: unearthing stuff hidden in the dungeons of Google’s results

October 4, 2012 Karen Blakeman 1 Comment

Fed up with seeing the same results from Google again and again? Wondering if that elusive document is buried somewhere at the bottom of Google’s 2,000,000 hits? Then get thee hence to Million Short (http://millionshort.com/). Million Short runs your search and then removes the most popular web sites from the results. Originally it removed the top 1 million, as its name suggests, but the default has changed to the top 10,000. The principle remains the same, though: exclude the more popular sites and you could uncover a real gem. The page that best answers your question might not be well optimised for search engines or might cover a topic that is so “niche” that it never makes it into the top results. Million Short does not say what it uses for search results or how it determines what are the most popular web sites. According to Webmonkey “Sanjay Arora, founder of Exponential Labs, tells Webmonkey that Million Short is using “the Bing API… augmented with some of our own data” for search results. What constitutes a “top site” in Million Short is determined by Alexa and Million Short’s own crawl data.” (http://www.webmonkey.com/2012/05/million-short-a-search-engine-for-the-very-long-tail/).

Using Million Short is straightforward. Type in your search and select how many sites you want to exclude (top 10K, top million, top 100). The results page includes a list of the sites that have been removed and you can opt to add one or more back in. You can also block a site using a link next to it in the results or click on “Boost!” so that pages from the site go to the top.

Million Short automatically tries to detect which country you are in but you can change it under “Manage Settings and Country”. I didn’t notice much difference when I changed countries but then most of the queries I pass through Million Short tend to be scientific or technical. On the same page you can manage sites that you have blocked, added or boosted.

Does it work? I would not use it instead of the existing major search engines such as Google, Bing or DuckDuckGo but as an additional tool to surface material that is not easily found in the likes of Google. As well as web search there are image and news searches, but I’m not convinced that I’d find those all that useful.

If you are interested in comparing Million Short with Google try Million Short It On at http://www.millionshortiton.com/index.html. I had several goes at this and most of the results were a draw. That is no surprise as the searches I ran were very specific and I wanted to see if Million Short would pull up additional information, which it did. Million Short won outright on a couple and Google on one. The Google win was by default because Million Short did not come up with anything for comparison (the search in question was biofuels public transport carbon emissions).

There are a number of techniques that you can use to improve Google results for example changing the order of the words in your search, Verbatim, filetype or Reading Level but I would also recommend trying Million Short. The results should at least be different and may reveal vital information for your research.

Google, Search Strategies

Yet another irritating Google feature

July 20, 2012 Karen Blakeman 4 Comments

There was a time when Google would aggregate pages from the same website in your search results. There might be just a couple of entries for the site with a “More from….” link next to the result.

Alternatively you might see a mini sitemap:

This has the advantage that you are not swamped with results from a single website but are given instead a variety of options that might provide you with a better answer to your question.

Not any more.

You may have noticed that multiple entries from single websites have started appearing in your results. For example, rather than just one Wikipedia entry you see 4, 5, 6 or even more. On the other hand, you might not have noticed anything at all. Some of my colleagues are seeing this and some are not. Google tests new features and algorithms on a small percentage of its users to see how they react so new or test features are not seen by everyone (see How Google makes improvements to its search algorithm – YouTube http://www.youtube.com/watch?v=J5RZOU6vK4Q). As far as I’m concerned this particular “improvement” is a disaster.

I was running a very general search on the use of biofuels by public transport in the UK. I just want to get an idea of some of the issues that were being discussed before refining my search and went, by default, to Google. My first screen had nothing but results from the UK government Department for Transport (DfT).

I scrolled down and saw more DfT pages. I scrolled down further and yet MORE dft pages. OK, Google, so dft.gov.uk is a good place for me to look at biofuels in public transport. I get the message. STOP! There were 27 DfT pages in total flooding the top of my results page, which I have set to display 100 entries at a time. Creeping in at number 28 came the Guardian with 5 results.

The Friends of the Earth website had 7 results, and then at last I started to see more variety in my results at around number 40, but still with a lot of repetition.

Google may think that the DfT is a very important source of information on the topic but I want to decide whether or not to explore more of a particular site. Spamming my results list annoys me and makes me want to go elsewhere. So I did.

DuckDuckGo (http://www.duckduckgo.com/) is my main Google alternative and it came up with a decent and varied set of results without repetition, hesitation or deviation.

Bing (http://www.bing.com/) and Yandex (http://www.yandex.com/) came up with similar, non-repetitive results.

Blekko (http://www.blekko.com/) came up with some interesting alternative pages for me to consider. These would not have been that useful to me in the earlier stages of my research but this test confirmed my feeling that Blekko is good at pulling up information that explores more than the mainstream issues.

If you want to stay with Google how do you deal with multiple listings of sites? The most obvious approach would be to incorporate a ‘-site:’ command in your search, for example:

biofuels public transport -site:dft.gov.uk

If you are conducting in depth research and are likely to be running many variations on a search, incorporating ‘-site:’ each time can become a chore. Google’s own browser Chrome has a Personal Blocklist extension that enables you to block selected sites from results (https://chrome.google.com/webstore/detail/nolijncfnkgaikbjbdaogikpmpbdcdef). Once installed a block link appears next to each entry in your results. Click on the link to block the site from all future results. A message appears at the bottom of searches that would normally contain pages from the blocked site warning you about exclusions.

The ‘show’ link displays and highlights the previously blocked pages and offers an option to unblock them.

Neither the -site: option nor the Blocklist approach should be necessary. There was nothing wrong with the previous ways of offering additional pages from a site in search results. It wasn’t broke but Google did break it by trying to fix it. For me, there are now several Google alternatives that produce quality results and with less irritation. I shall be using them more in future.

Inforum, Presentations, Search Engines, Search Strategies

Search gets personal and social

May 29, 2012 Karen Blakeman

My INFORUM 2012 presentation on “Search gets personal and social” is available on authorSTREAM at http://www.authorstream.com/Presentation/karenblakeman-1431533-search-gets-personal-and-social/

It is also available temporarily at http://www.rba.co.uk/as/

A paper is also available on the INFORUM web site at http://www.inforum.cz/en/proceedings. It covers much of what I said but bear in mind it was written a few weeks beforehand and the presentation was updated with new developments the night before I gave the talk.

Karen Blakeman's Blog